undefined step-generate Cluster name generated: nas-412-103025 Is nightly?: false Image repository: devtools_gitops Result path: CI/z-stream-test-pipeline/241030-110234 Result path URL: http://uploader.devtools-gitops--pipeline.svc.cluster.local:8080/CI/z-stream-test-pipeline/241030-110234/ undefined step-create-cluster-dir Directory with cluster info exists, logging in WARNING: Using insecure TLS client config. Setting this option is not supported! Login successful. You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". Welcome! See 'oc help' to get started. undefined step-clone + '[' false = true ']' + '[' false = true ']' + '[' false = true ']' + CHECKOUT_DIR=/workspace/output/ + '[' false = true ']' + test -z '' + test -z '' + test -z '' + git config --global --add safe.directory /workspace/output + /ko-app/git-init -url=https://gitlab.cee.redhat.com/gitops/plumbing-gitops.git -revision= -refspec= -path=/workspace/output/ -sslVerify=false -submodules=true -depth=1 -sparseCheckoutDirectories= {"level":"info","ts":1730286164.6827724,"caller":"git/git.go:176","msg":"Successfully cloned https://gitlab.cee.redhat.com/gitops/plumbing-gitops.git @ a4859e5c72de7caa0f0b3995cae8d9b545ba0ba7 (grafted, HEAD) in path /workspace/output/"} {"level":"info","ts":1730286164.7886176,"caller":"git/git.go:215","msg":"Successfully initialized and updated submodules in path /workspace/output/"} + cd /workspace/output/ ++ git rev-parse HEAD + RESULT_SHA=a4859e5c72de7caa0f0b3995cae8d9b545ba0ba7 + EXIT_CODE=0 + '[' 0 '!=' 0 ']' ++ git log -1 --pretty=%ct + RESULT_COMMITTER_DATE=1729838808 + printf %s 1729838808 + printf %s a4859e5c72de7caa0f0b3995cae8d9b545ba0ba7 + printf %s https://gitlab.cee.redhat.com/gitops/plumbing-gitops.git undefined step-branch-version gitops version latest is Present undefined step-create-catalog-source Logging in as kubeadmin user WARNING: Using insecure TLS client config. Setting this option is not supported! Login successful. You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". Welcome! See 'oc help' to get started. NAME DISPLAY TYPE PUBLISHER AGE redhat-operators Red Hat Operators grpc Red Hat 13d pod/redhat-operators-sjkq5 condition met Operator will be installed from default catalog source redhat-operators passed undefined step-test-argo-rollouts + RESULT=/install-dir/nas-412-result.txt Logging in as kubeadmin user + trap update_result EXIT + echo -e 'Logging in as kubeadmin user' ++ cat /install-dir/nas-412/auth/kubeadmin-password ++ cat /install-dir/nas-412/auth/api-url + oc login -u kubeadmin -p JFr6Y-6D72V-jdUqy-g9CgL https://api.nas-412.ocp-gitops-qe.com:6443 --insecure-skip-tls-verify=true WARNING: Using insecure TLS client config. Setting this option is not supported! Login successful. You have access to 70 projects, the list has been suppressed. You can list all projects with 'oc projects' Using project "default". Welcome! See 'oc help' to get started. ++ echo true ++ tr '[:upper:]' '[:lower:]' + [[ true == \f\a\l\s\e ]] + exit_code=0 + failed=0 ++ oc get deployment openshift-gitops-operator-controller-manager -n openshift-gitops-operator -o 'jsonpath={.metadata.namespace}' --ignore-not-found + openshift_gitops_operator= + operator_namespace=openshift-operators + oc patch subscription openshift-gitops-operator -n openshift-operators --type merge --patch '{"spec": {"config": {"env": [{"name": "CLUSTER_SCOPED_ARGO_ROLLOUTS_NAMESPACES", "value": "argo-rollouts,test-rom-ns-1,rom-ns-1"}]}}}' subscription.operators.coreos.com/openshift-gitops-operator patched (no change) ++ mktemp -d + ROLLOUTS_TMP_DIR=/tmp/tmp.hNjWJyuQmo + cd /tmp/tmp.hNjWJyuQmo + git clone https://github.com/argoproj-labs/argo-rollouts-manager Cloning into 'argo-rollouts-manager'... + cd /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager + TARGET_ROLLOUT_MANAGER_COMMIT=e054d35fd0d9dad6b9ae26efa60f43cadc194b4b + TARGET_OPENSHIFT_ROUTE_ROLLOUT_PLUGIN_COMMIT=8b4125a7f9ecffb0247df91a4c890f88c0c523b7 + git checkout e054d35fd0d9dad6b9ae26efa60f43cadc194b4b Note: switching to 'e054d35fd0d9dad6b9ae26efa60f43cadc194b4b'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at e054d35 Backport cluster/namespace-scoped RBAC tests to 0.0.3 + make test-e2e NAMESPACE_SCOPED_ARGO_ROLLOUTS=false hack/run-rollouts-manager-e2e-tests.sh + '[' false == true ']' + go test -v -p=1 -timeout=30m -race -count=1 -coverprofile=coverage.out ./tests/e2e/cluster-scoped go: downloading sigs.k8s.io/controller-runtime v0.16.3 go: downloading go.uber.org/zap v1.25.0 go: downloading github.com/onsi/gomega v1.27.10 go: downloading github.com/onsi/ginkgo/v2 v2.11.0 go: downloading github.com/coreos/prometheus-operator v0.40.0 go: downloading k8s.io/api v0.28.3 go: downloading k8s.io/apiextensions-apiserver v0.28.3 go: downloading k8s.io/apimachinery v0.28.3 go: downloading k8s.io/client-go v0.28.3 go: downloading github.com/go-logr/logr v1.2.4 go: downloading gopkg.in/yaml.v2 v2.4.0 go: downloading go.uber.org/multierr v1.11.0 go: downloading github.com/google/go-cmp v0.5.9 go: downloading github.com/gogo/protobuf v1.3.2 go: downloading k8s.io/utils v0.0.0-20230406110748-d93618cff8a2 go: downloading github.com/go-logr/zapr v1.2.4 go: downloading github.com/evanphx/json-patch/v5 v5.6.0 go: downloading github.com/evanphx/json-patch v5.6.0+incompatible go: downloading github.com/google/gofuzz v1.2.0 go: downloading k8s.io/klog/v2 v2.100.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.2.3 go: downloading sigs.k8s.io/yaml v1.3.0 go: downloading golang.org/x/net v0.23.0 go: downloading gopkg.in/yaml.v3 v3.0.1 go: downloading golang.org/x/sys v0.18.0 go: downloading sigs.k8s.io/json v0.0.0-20221116044647-bc3834ca7abd go: downloading k8s.io/kube-openapi v0.0.0-20230717233707-2695361300d9 go: downloading github.com/pkg/errors v0.9.1 go: downloading gopkg.in/inf.v0 v0.9.1 go: downloading github.com/json-iterator/go v1.1.12 go: downloading gomodules.xyz/jsonpatch/v2 v2.4.0 go: downloading k8s.io/component-base v0.28.3 go: downloading golang.org/x/text v0.14.0 go: downloading golang.org/x/exp v0.0.0-20220827204233-334a2380cb91 go: downloading github.com/prometheus/client_golang v1.16.0 go: downloading github.com/davecgh/go-spew v1.1.1 go: downloading github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd go: downloading github.com/modern-go/reflect2 v1.0.2 go: downloading github.com/google/gnostic-models v0.6.8 go: downloading github.com/spf13/pflag v1.0.5 go: downloading golang.org/x/term v0.18.0 go: downloading github.com/imdario/mergo v0.3.13 go: downloading github.com/golang/protobuf v1.5.3 go: downloading github.com/google/uuid v1.3.0 go: downloading github.com/fsnotify/fsnotify v1.6.0 go: downloading github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da go: downloading golang.org/x/time v0.3.0 go: downloading golang.org/x/oauth2 v0.10.0 go: downloading google.golang.org/protobuf v1.33.0 go: downloading github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 go: downloading github.com/go-openapi/swag v0.22.3 go: downloading github.com/go-openapi/jsonreference v0.20.2 go: downloading github.com/emicklei/go-restful/v3 v3.11.0 go: downloading github.com/prometheus/client_model v0.4.0 go: downloading github.com/prometheus/common v0.44.0 go: downloading github.com/beorn7/perks v1.0.1 go: downloading github.com/cespare/xxhash/v2 v2.2.0 go: downloading github.com/prometheus/procfs v0.10.1 go: downloading github.com/mailru/easyjson v0.7.7 go: downloading github.com/go-openapi/jsonpointer v0.19.6 go: downloading github.com/josharian/intern v1.0.0 go: downloading github.com/matttproud/golang_protobuf_extensions v1.0.4 === RUN TestClusterScoped Running Suite: Cluster-scoped Suite - /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/cluster-scoped ======================================================================================================== Random Seed: 1730286501 Will run 6 of 6 specs HavePhase: expected: Available actual: HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Available HaveSuccessCondition: expected: {Reconciled True 0 0001-01-01 00:00:00 +0000 UTC Success } actual: {Reconciled True 0 2024-10-30 11:08:37 +0000 UTC Success } Object exists in ExistByName: argo-rollouts Object exists in ExistByName: argo-rollouts ------------------------------ • [FAILED] [31.934 seconds] Cluster-scoped RolloutManager tests Testing cluster-scoped RolloutManager behaviour [It] After creating cluster-scoped RolloutManager in one namespace, operator should create appropriate K8s resources and watch argo rollouts CR in another namespace. /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/cluster-scoped/cluster_scoped_rollouts_test.go:49 Timeline >> STEP: Create cluster-scoped RolloutManager in default namespace. @ 10/30/24 11:08:37.207 STEP: Verify that RolloutManager is successfully created. @ 10/30/24 11:08:37.24 STEP: Verify that Status.Condition is having success condition. @ 10/30/24 11:08:53.748 STEP: Verify that expected resources are created. @ 10/30/24 11:08:53.775 STEP: Verify that ServiceAccount is created. @ 10/30/24 11:08:53.775 STEP: Verify that ServiceAccount has correct labels and annotations. @ 10/30/24 11:08:53.799 STEP: Verify that argo-rollout ClusterRoles is created. @ 10/30/24 11:08:53.799 STEP: Verify that ClusterRole has correct labels and annotations. @ 10/30/24 11:08:53.819 STEP: Verify that ClusterRole has correct policy rules. @ 10/30/24 11:08:53.819 [FAILED] in [It] - /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/test_utils.go:215 @ 10/30/24 11:08:53.828 << Timeline [FAILED] Expected <[]v1.PolicyRule | len:23, cap:34>: [ { Verbs: ["get", "list", "watch", "update", "patch"], APIGroups: ["argoproj.io"], Resources: [ "rollouts", "rollouts/status", "rollouts/finalizers", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["argoproj.io"], Resources: [ "analysisruns", "analysisruns/finalizers", "experiments", "experiments/finalizers", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: ["argoproj.io"], Resources: [ "analysistemplates", "clusteranalysistemplates", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["apps"], Resources: ["replicasets"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch", "update", "patch"], APIGroups: ["", "apps"], Resources: ["deployments", "podtemplates"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch", "patch", "create", "delete"], APIGroups: [""], Resources: ["services"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "update"], APIGroups: [ "coordination.k8s.io", ], Resources: ["leases"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: [""], Resources: ["secrets", "configmaps"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["list", "update", "watch"], APIGroups: [""], Resources: ["pods"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create"], APIGroups: [""], Resources: ["pods/eviction"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "update", "patch"], APIGroups: [""], Resources: ["events"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "patch"], APIGroups: ["networking.k8s.io", "extensions"], Resources: ["ingresses"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["batch"], Resources: ["jobs"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["watch", "get", "update", "patch", "list"], APIGroups: [ "networking.istio.io", ], Resources: ["virtualservices", "destinationrules"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "watch", "get", "update", "patch"], APIGroups: ["split.smi-spec.io"], Resources: ["trafficsplits"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "watch", "get", "update", "list", "delete"], APIGroups: ["getambassador.io", "x.getambassador.io"], Resources: ["mappings", "... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output to consist of <[]v1.PolicyRule | len:23, cap:23>: [ { Verbs: ["get", "list", "watch", "update", "patch"], APIGroups: ["argoproj.io"], Resources: [ "rollouts", "rollouts/status", "rollouts/finalizers", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["argoproj.io"], Resources: [ "analysisruns", "analysisruns/finalizers", "experiments", "experiments/finalizers", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: ["argoproj.io"], Resources: [ "analysistemplates", "clusteranalysistemplates", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["apps"], Resources: ["replicasets"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: ["", "apps"], Resources: ["deployments", "podtemplates"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch", "patch", "create", "delete"], APIGroups: [""], Resources: ["services"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "update"], APIGroups: [ "coordination.k8s.io", ], Resources: ["leases"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: [""], Resources: ["secrets", "configmaps"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["list", "update", "watch"], APIGroups: [""], Resources: ["pods"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create"], APIGroups: [""], Resources: ["pods/eviction"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "update", "patch"], APIGroups: [""], Resources: ["events"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "patch"], APIGroups: ["networking.k8s.io", "extensions"], Resources: ["ingresses"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["batch"], Resources: ["jobs"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["watch", "get", "update", "patch", "list"], APIGroups: [ "networking.istio.io", ], Resources: ["virtualservices", "destinationrules"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "watch", "get", "update", "patch"], APIGroups: ["split.smi-spec.io"], Resources: ["trafficsplits"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "watch", "get", "update", "list", "delete"], APIGroups: ["getambassador.io", "x.getambassador.io"], Resources: ["mappings", "ambassadormappings"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output the missing elements were <[]v1.PolicyRule | len:2, cap:2>: [ { Verbs: ["get", "list", "watch"], APIGroups: ["", "apps"], Resources: ["deployments", "podtemplates"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["watch", "get", "update"], APIGroups: [ "traefik.containo.us", ], Resources: ["traefikservices"], ResourceNames: nil, NonResourceURLs: nil, }, ] the extra elements were <[]v1.PolicyRule | len:2, cap:2>: [ { Verbs: ["get", "list", "watch", "update", "patch"], APIGroups: ["", "apps"], Resources: ["deployments", "podtemplates"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["watch", "get", "update"], APIGroups: [ "traefik.containo.us", "traefik.io", ], Resources: ["traefikservices"], ResourceNames: nil, NonResourceURLs: nil, }, ] In [It] at: /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/test_utils.go:215 @ 10/30/24 11:08:53.828 ------------------------------ HavePhase: expected: Available actual: HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Available HaveSuccessCondition: expected: {Reconciled True 0 0001-01-01 00:00:00 +0000 UTC Success } actual: {Reconciled True 0 2024-10-30 11:09:09 +0000 UTC Success } Object exists in ExistByName: argo-rollouts Object exists in ExistByName: argo-rollouts • [FAILED] [31.885 seconds] Cluster-scoped RolloutManager tests Testing cluster-scoped RolloutManager behaviour [It] After creating cluster-scoped RolloutManager in a namespace, operator should create appropriate K8s resources and watch argo rollouts CR in multiple namespace. /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/cluster-scoped/cluster_scoped_rollouts_test.go:90 Timeline >> STEP: Create a namespace for RolloutManager. @ 10/30/24 11:09:09.106 STEP: Create cluster-scoped RolloutManager. @ 10/30/24 11:09:09.137 STEP: Verify that RolloutManager is successfully created. @ 10/30/24 11:09:09.166 STEP: Verify that Status.Condition is having success condition. @ 10/30/24 11:09:25.646 STEP: Verify that expected resources are created. @ 10/30/24 11:09:25.672 STEP: Verify that ServiceAccount is created. @ 10/30/24 11:09:25.672 STEP: Verify that ServiceAccount has correct labels and annotations. @ 10/30/24 11:09:25.682 STEP: Verify that argo-rollout ClusterRoles is created. @ 10/30/24 11:09:25.682 STEP: Verify that ClusterRole has correct labels and annotations. @ 10/30/24 11:09:25.703 STEP: Verify that ClusterRole has correct policy rules. @ 10/30/24 11:09:25.703 [FAILED] in [It] - /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/test_utils.go:215 @ 10/30/24 11:09:25.715 << Timeline [FAILED] Expected <[]v1.PolicyRule | len:23, cap:34>: [ { Verbs: ["get", "list", "watch", "update", "patch"], APIGroups: ["argoproj.io"], Resources: [ "rollouts", "rollouts/status", "rollouts/finalizers", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["argoproj.io"], Resources: [ "analysisruns", "analysisruns/finalizers", "experiments", "experiments/finalizers", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: ["argoproj.io"], Resources: [ "analysistemplates", "clusteranalysistemplates", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["apps"], Resources: ["replicasets"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch", "update", "patch"], APIGroups: ["", "apps"], Resources: ["deployments", "podtemplates"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch", "patch", "create", "delete"], APIGroups: [""], Resources: ["services"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "update"], APIGroups: [ "coordination.k8s.io", ], Resources: ["leases"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: [""], Resources: ["secrets", "configmaps"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["list", "update", "watch"], APIGroups: [""], Resources: ["pods"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create"], APIGroups: [""], Resources: ["pods/eviction"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "update", "patch"], APIGroups: [""], Resources: ["events"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "patch"], APIGroups: ["networking.k8s.io", "extensions"], Resources: ["ingresses"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["batch"], Resources: ["jobs"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["watch", "get", "update", "patch", "list"], APIGroups: [ "networking.istio.io", ], Resources: ["virtualservices", "destinationrules"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "watch", "get", "update", "patch"], APIGroups: ["split.smi-spec.io"], Resources: ["trafficsplits"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "watch", "get", "update", "list", "delete"], APIGroups: ["getambassador.io", "x.getambassador.io"], Resources: ["mappings", "... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output to consist of <[]v1.PolicyRule | len:23, cap:23>: [ { Verbs: ["get", "list", "watch", "update", "patch"], APIGroups: ["argoproj.io"], Resources: [ "rollouts", "rollouts/status", "rollouts/finalizers", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["argoproj.io"], Resources: [ "analysisruns", "analysisruns/finalizers", "experiments", "experiments/finalizers", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: ["argoproj.io"], Resources: [ "analysistemplates", "clusteranalysistemplates", ], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["apps"], Resources: ["replicasets"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: ["", "apps"], Resources: ["deployments", "podtemplates"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch", "patch", "create", "delete"], APIGroups: [""], Resources: ["services"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "update"], APIGroups: [ "coordination.k8s.io", ], Resources: ["leases"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["get", "list", "watch"], APIGroups: [""], Resources: ["secrets", "configmaps"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["list", "update", "watch"], APIGroups: [""], Resources: ["pods"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create"], APIGroups: [""], Resources: ["pods/eviction"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "update", "patch"], APIGroups: [""], Resources: ["events"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "patch"], APIGroups: ["networking.k8s.io", "extensions"], Resources: ["ingresses"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "get", "list", "watch", "update", "patch", "delete"], APIGroups: ["batch"], Resources: ["jobs"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["watch", "get", "update", "patch", "list"], APIGroups: [ "networking.istio.io", ], Resources: ["virtualservices", "destinationrules"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "watch", "get", "update", "patch"], APIGroups: ["split.smi-spec.io"], Resources: ["trafficsplits"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["create", "watch", "get", "update", "list", "delete"], APIGroups: ["getambassador.io", "x.getambassador.io"], Resources: ["mappings", "ambassadormappings"... Gomega truncated this representation as it exceeds 'format.MaxLength'. Consider having the object provide a custom 'GomegaStringer' representation or adjust the parameters in Gomega's 'format' package. Learn more here: https://onsi.github.io/gomega/#adjusting-output the missing elements were <[]v1.PolicyRule | len:2, cap:2>: [ { Verbs: ["get", "list", "watch"], APIGroups: ["", "apps"], Resources: ["deployments", "podtemplates"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["watch", "get", "update"], APIGroups: [ "traefik.containo.us", ], Resources: ["traefikservices"], ResourceNames: nil, NonResourceURLs: nil, }, ] the extra elements were <[]v1.PolicyRule | len:2, cap:2>: [ { Verbs: ["get", "list", "watch", "update", "patch"], APIGroups: ["", "apps"], Resources: ["deployments", "podtemplates"], ResourceNames: nil, NonResourceURLs: nil, }, { Verbs: ["watch", "get", "update"], APIGroups: [ "traefik.containo.us", "traefik.io", ], Resources: ["traefikservices"], ResourceNames: nil, NonResourceURLs: nil, }, ] In [It] at: /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/test_utils.go:215 @ 10/30/24 11:09:25.715 ------------------------------ HavePhase: expected: Available actual: HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Available HaveSuccessCondition: expected: {Reconciled True 0 0001-01-01 00:00:00 +0000 UTC Success } actual: {Reconciled True 0 2024-10-30 11:09:51 +0000 UTC Success } Object exists in ExistByName: rollout-bluegreen-active Object exists in ExistByName: rollout-bluegreen-preview HavePhase: expected: Failure actual: Failure HaveCondition: expected: {Reconciled False 0 0001-01-01 00:00:00 +0000 UTC InvalidRolloutManagerScope when Subscription has environment variable NAMESPACE_SCOPED_ARGO_ROLLOUTS set to False, there may not exist any namespace-scoped RolloutManagers: only a single cluster-scoped RolloutManager is supported} actual: {Reconciled False 0 2024-10-30 11:10:10 +0000 UTC InvalidRolloutManagerScope when Subscription has environment variable NAMESPACE_SCOPED_ARGO_ROLLOUTS set to False, there may not exist any namespace-scoped RolloutManagers: only a single cluster-scoped RolloutManager is supported} Object exists in ExistByName: rollout-bluegreen-active Object exists in ExistByName: rollout-bluegreen-preview HavePhase: expected: Available actual: Available HaveSuccessCondition: expected: {Reconciled True 0 0001-01-01 00:00:00 +0000 UTC Success } actual: {Reconciled True 0 2024-10-30 11:09:51 +0000 UTC Success } Object exists in ExistByName: rollout-bluegreen-active Object exists in ExistByName: rollout-bluegreen-preview •HavePhase: expected: Available actual: HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Available HaveSuccessCondition: expected: {Reconciled True 0 0001-01-01 00:00:00 +0000 UTC Success } actual: {Reconciled True 0 2024-10-30 11:10:57 +0000 UTC Success } Object exists in ExistByName: rollout-bluegreen-active Object exists in ExistByName: rollout-bluegreen-preview HavePhase: expected: Failure actual: Failure HaveCondition: expected: {Reconciled False 0 0001-01-01 00:00:00 +0000 UTC MultipleClusterScopedRolloutManager when there exists a cluster-scoped RolloutManager on the cluster, there may not exist another: only a single cluster-scoped RolloutManager is supported} actual: {Reconciled False 0 2024-10-30 11:11:17 +0000 UTC MultipleClusterScopedRolloutManager when there exists a cluster-scoped RolloutManager on the cluster, there may not exist another: only a single cluster-scoped RolloutManager is supported} Object exists in ExistByName: rollout-bluegreen-active Object exists in ExistByName: rollout-bluegreen-preview HavePhase: expected: Failure actual: Failure HaveCondition: expected: {Reconciled False 0 0001-01-01 00:00:00 +0000 UTC MultipleClusterScopedRolloutManager when there exists a cluster-scoped RolloutManager on the cluster, there may not exist another: only a single cluster-scoped RolloutManager is supported} actual: {Reconciled False 0 2024-10-30 11:11:17 +0000 UTC MultipleClusterScopedRolloutManager when there exists a cluster-scoped RolloutManager on the cluster, there may not exist another: only a single cluster-scoped RolloutManager is supported} Object exists in ExistByName: rollout-bluegreen-active Object exists in ExistByName: rollout-bluegreen-preview •HavePhase: expected: Failure actual: Failure HaveCondition: expected: {Reconciled False 0 0001-01-01 00:00:00 +0000 UTC InvalidRolloutManagerNamespace Namespace is not specified in CLUSTER_SCOPED_ARGO_ROLLOUTS_NAMESPACES environment variable of Subscription resource. If you wish to install a cluster-scoped Argo Rollouts instance outside the default namespace, ensure it is defined in CLUSTER_SCOPED_ARGO_ROLLOUTS_NAMESPACES} actual: {Reconciled False 0 2024-10-30 11:12:09 +0000 UTC InvalidRolloutManagerNamespace Namespace is not specified in CLUSTER_SCOPED_ARGO_ROLLOUTS_NAMESPACES environment variable of Subscription resource. If you wish to install a cluster-scoped Argo Rollouts instance outside the default namespace, ensure it is defined in CLUSTER_SCOPED_ARGO_ROLLOUTS_NAMESPACES} •HavePhase: expected: Available actual: HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Pending HavePhase: expected: Available actual: Available ------------------------------ • [FAILED] [36.851 seconds] Cluster-scoped RolloutManager tests Backport of cluster/namespace-scoped RBAC tests to 0.0.3 when a namespace-scoped RolloutManager is installed into a namespace that previously contained a cluster-scoped RolloutManager, or vice versa [It] should cleanup any cluster/role/rolebinding resources that are present in the namespace, that do not match the current .spec.namespaceScoped value of the RolloutManager CR /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/cluster-scoped/cluster_scoped_rollouts_test.go:329 Timeline >> STEP: creating ClusterRole/Binding in the namespace-scoped case, and Role/Binding in the cluster-scoped case @ 10/30/24 11:12:29.756 STEP: creating RolloutManager and waiting for it to be available @ 10/30/24 11:12:29.796 STEP: verifying that in the cluster-scoped case, the namespace-scoped resources are deleted after reconciliation @ 10/30/24 11:12:46.312 [FAILED] in [It] - /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/cluster-scoped/cluster_scoped_rollouts_test.go:414 @ 10/30/24 11:12:46.323 << Timeline [FAILED] Expected failure, but got no error. In [It] at: /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/cluster-scoped/cluster_scoped_rollouts_test.go:414 @ 10/30/24 11:12:46.323 ------------------------------ Summarizing 3 Failures: [FAIL] Cluster-scoped RolloutManager tests Testing cluster-scoped RolloutManager behaviour [It] After creating cluster-scoped RolloutManager in one namespace, operator should create appropriate K8s resources and watch argo rollouts CR in another namespace. /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/test_utils.go:215 [FAIL] Cluster-scoped RolloutManager tests Testing cluster-scoped RolloutManager behaviour [It] After creating cluster-scoped RolloutManager in a namespace, operator should create appropriate K8s resources and watch argo rollouts CR in multiple namespace. /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/test_utils.go:215 [FAIL] Cluster-scoped RolloutManager tests Backport of cluster/namespace-scoped RBAC tests to 0.0.3 when a namespace-scoped RolloutManager is installed into a namespace that previously contained a cluster-scoped RolloutManager, or vice versa [It] should cleanup any cluster/role/rolebinding resources that are present in the namespace, that do not match the current .spec.namespaceScoped value of the RolloutManager CR /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/tests/e2e/cluster-scoped/cluster_scoped_rollouts_test.go:414 Ran 6 of 6 Specs in 264.433 seconds FAIL! -- 3 Passed | 3 Failed | 0 Pending | 0 Skipped --- FAIL: TestClusterScoped (264.44s) FAIL coverage: [no statements] FAIL github.com/argoproj-labs/argo-rollouts-manager/tests/e2e/cluster-scoped 264.543s FAIL make: *** [Makefile:120: test-e2e] Error 1 + exit_code=2 + [[ 2 != 0 ]] + echo 2 + tee /tekton/results/result 2 + failed=2 + exit_code=0 + kubectl delete rolloutmanagers --all -n test-rom-ns-1 No resources found + cd /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager + SKIP_RUN_STEP=true + hack/run-upstream-argo-rollouts-e2e-tests.sh + set -e + trap cleanup EXIT +++ dirname -- hack/run-upstream-argo-rollouts-e2e-tests.sh ++ cd -- hack ++ pwd + SCRIPT_DIR=/tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/hack ++ mktemp -d + TMP_DIR=/tmp/tmp.58kGTG6WqB + cd /tmp/tmp.58kGTG6WqB + git clone https://github.com/argoproj/argo-rollouts Cloning into 'argo-rollouts'... + cd argo-rollouts + git checkout v1.6.6 Note: switching to 'v1.6.6'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 737ca89b4 fix: stuck rollout when 2nd deployment happens before 1st finishes (#3354) + go mod tidy go: downloading github.com/blang/semver v3.5.1+incompatible go: downloading k8s.io/kube-openapi v0.0.0-20221012153701-172d655c2280 go: downloading k8s.io/apimachinery v0.25.8 go: downloading k8s.io/apiextensions-apiserver v0.25.8 go: downloading k8s.io/cli-runtime v0.25.8 go: downloading k8s.io/client-go v0.25.8 go: downloading github.com/stretchr/testify v1.8.4 go: downloading k8s.io/klog/v2 v2.80.1 go: downloading github.com/sirupsen/logrus v1.9.3 go: downloading github.com/aws/aws-sdk-go-v2/config v1.19.1 go: downloading github.com/aws/aws-sdk-go-v2/service/cloudwatch v1.27.2 go: downloading github.com/argoproj/notifications-engine v0.4.1-0.20231011160156-2d2d1a75dbee go: downloading github.com/newrelic/newrelic-client-go v1.1.0 go: downloading github.com/aws/aws-sdk-go-v2 v1.21.2 go: downloading github.com/argoproj/pkg v0.13.6 go: downloading github.com/servicemeshinterface/smi-sdk-go v0.5.0 go: downloading github.com/grpc-ecosystem/grpc-gateway v1.16.0 go: downloading k8s.io/code-generator v0.25.8 go: downloading k8s.io/api v0.25.8 go: downloading github.com/hashicorp/go-plugin v1.4.10 go: downloading k8s.io/component-base v0.25.8 go: downloading k8s.io/kubernetes v1.25.8 go: downloading github.com/influxdata/influxdb-client-go/v2 v2.12.3 go: downloading github.com/machinebox/graphql v0.2.2 go: downloading github.com/spaceapegames/go-wavefront v1.8.1 go: downloading github.com/tj/assert v0.0.3 go: downloading google.golang.org/genproto/googleapis/api v0.0.0-20230706204954-ccb25ca9f130 go: downloading google.golang.org/genproto v0.0.0-20230706204954-ccb25ca9f130 go: downloading google.golang.org/grpc v1.57.0 go: downloading github.com/prometheus/common v0.42.0 go: downloading github.com/prometheus/common/sigv4 v0.1.0 go: downloading google.golang.org/protobuf v1.31.0 go: downloading github.com/spf13/cobra v1.7.0 go: downloading github.com/juju/ansiterm v1.0.0 go: downloading k8s.io/kubectl v0.25.8 go: downloading github.com/aws/aws-sdk-go-v2/service/elasticloadbalancingv2 v1.21.0 go: downloading github.com/soheilhy/cmux v0.1.5 go: downloading k8s.io/apiserver v0.25.8 go: downloading github.com/antonmedv/expr v1.15.3 go: downloading github.com/mitchellh/mapstructure v1.5.0 go: downloading github.com/bombsimon/logrusr/v4 v4.0.0 go: downloading github.com/valyala/fasttemplate v1.2.2 go: downloading github.com/golang/mock v1.6.0 go: downloading github.com/prometheus/client_model v0.3.0 go: downloading golang.org/x/sys v0.10.0 go: downloading github.com/pmezard/go-difflib v1.0.0 go: downloading github.com/stretchr/objx v0.5.0 go: downloading github.com/aws/aws-sdk-go-v2/credentials v1.13.43 go: downloading github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.13.13 go: downloading github.com/aws/aws-sdk-go-v2/internal/ini v1.3.45 go: downloading github.com/aws/aws-sdk-go-v2/service/sso v1.15.2 go: downloading github.com/aws/aws-sdk-go-v2/service/ssooidc v1.17.3 go: downloading github.com/aws/aws-sdk-go-v2/service/sts v1.23.2 go: downloading github.com/aws/smithy-go v1.15.0 go: downloading github.com/aws/aws-sdk-go-v2/internal/configsources v1.1.43 go: downloading github.com/jmespath/go-jmespath v0.4.0 go: downloading github.com/go-openapi/jsonreference v0.20.0 go: downloading github.com/go-openapi/swag v0.21.1 go: downloading github.com/google/gnostic v0.6.9 go: downloading github.com/golang/glog v1.1.0 go: downloading k8s.io/gengo v0.0.0-20220902162205-c0856e24416d go: downloading github.com/hashicorp/go-hclog v0.14.1 go: downloading github.com/hashicorp/yamux v0.0.0-20180604194846-3520598351bb go: downloading github.com/mitchellh/go-testing-interface v1.0.0 go: downloading github.com/oklog/run v1.0.0 go: downloading golang.org/x/tools v0.6.0 go: downloading golang.org/x/net v0.12.0 go: downloading github.com/deepmap/oapi-codegen v1.11.0 go: downloading github.com/matryer/is v1.4.0 go: downloading github.com/felixge/httpsnoop v1.0.3 go: downloading google.golang.org/genproto/googleapis/rpc v0.0.0-20230711160842-782d3b101e98 go: downloading github.com/Azure/go-autorest/autorest v0.11.27 go: downloading github.com/Azure/go-autorest v14.2.0+incompatible go: downloading github.com/Azure/go-autorest/autorest/adal v0.9.20 go: downloading github.com/aws/aws-sdk-go v1.44.116 go: downloading github.com/elazarl/goproxy v0.0.0-20220417044921-416226498f94 go: downloading github.com/jhump/protoreflect v1.6.0 go: downloading golang.org/x/term v0.10.0 go: downloading github.com/emicklei/go-restful/v3 v3.10.1 go: downloading gopkg.in/check.v1 v1.0.0-20201130134442-10cb98267c6c go: downloading github.com/inconshreveable/mousetrap v1.1.0 go: downloading github.com/mattn/go-colorable v0.1.12 go: downloading github.com/mattn/go-isatty v0.0.14 go: downloading github.com/valyala/bytebufferpool v1.0.0 go: downloading github.com/PagerDuty/go-pagerduty v1.7.0 go: downloading github.com/RocketChat/Rocket.Chat.Go.SDK v0.0.0-20220708192748-b73dcb041214 go: downloading github.com/aws/aws-sdk-go-v2/service/sqs v1.20.0 go: downloading github.com/bradleyfalzon/ghinstallation/v2 v2.5.0 go: downloading github.com/go-telegram-bot-api/telegram-bot-api/v5 v5.5.1 go: downloading github.com/google/go-github/v41 v41.0.0 go: downloading github.com/gregdel/pushover v1.2.1 go: downloading github.com/opsgenie/opsgenie-go-sdk-v2 v1.2.13 go: downloading github.com/slack-go/slack v0.12.2 go: downloading github.com/spf13/cast v1.5.1 go: downloading github.com/whilp/git-urls v0.0.0-20191001220047-6db9661140c0 go: downloading gomodules.xyz/notify v0.1.1 go: downloading google.golang.org/api v0.132.0 go: downloading github.com/Masterminds/sprig/v3 v3.2.3 go: downloading sigs.k8s.io/json v0.0.0-20220713155537-f223a00ba0e2 go: downloading github.com/liggitt/tabwriter v0.0.0-20181228230101-89fcab3d43de go: downloading golang.org/x/text v0.11.0 go: downloading sigs.k8s.io/kustomize/api v0.12.1 go: downloading sigs.k8s.io/kustomize/kyaml v0.13.9 go: downloading github.com/gregjones/httpcache v0.0.0-20190611155906-901d90724c79 go: downloading github.com/peterbourgon/diskv v2.0.1+incompatible go: downloading github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.9.37 go: downloading github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.4.37 go: downloading github.com/google/go-querystring v1.1.0 go: downloading github.com/hashicorp/go-retryablehttp v0.7.1 go: downloading github.com/tomnomnom/linkheader v0.0.0-20180905144013-02ca5825eb80 go: downloading github.com/valyala/fastjson v1.6.3 go: downloading github.com/go-openapi/jsonpointer v0.19.5 go: downloading github.com/jmespath/go-jmespath/internal/testify v1.5.1 go: downloading github.com/ghodss/yaml v1.0.1-0.20190212211648-25d852aebe32 go: downloading github.com/fatih/color v1.7.0 go: downloading github.com/blang/semver/v4 v4.0.0 go: downloading github.com/influxdata/line-protocol v0.0.0-20210922203350-b1ad95c89adf go: downloading github.com/mwitkow/go-conntrack v0.0.0-20190716064945-2f068394615f go: downloading github.com/Azure/go-autorest/tracing v0.6.0 go: downloading github.com/Azure/go-autorest/logger v0.2.1 go: downloading github.com/Azure/go-autorest/autorest/mocks v0.4.2 go: downloading github.com/Azure/go-autorest/autorest/date v0.3.0 go: downloading github.com/golang-jwt/jwt/v4 v4.5.0 go: downloading golang.org/x/crypto v0.11.0 go: downloading cloud.google.com/go/compute/metadata v0.2.3 go: downloading cloud.google.com/go/compute v1.20.1 go: downloading google.golang.org/appengine v1.6.7 go: downloading github.com/kr/pretty v0.3.1 go: downloading github.com/onsi/ginkgo/v2 v2.1.6 go: downloading github.com/onsi/gomega v1.20.1 go: downloading github.com/lunixbochs/vtclean v1.0.0 go: downloading github.com/google/go-github/v53 v53.0.0 go: downloading k8s.io/component-helpers v0.25.8 go: downloading github.com/exponent-io/jsonpath v0.0.0-20210407135951-1de76d718b3f go: downloading github.com/MakeNowJust/heredoc v1.0.0 go: downloading github.com/mitchellh/go-wordwrap v1.0.1 go: downloading github.com/russross/blackfriday v1.6.0 go: downloading github.com/chai2010/gettext-go v1.0.2 go: downloading github.com/frankban/quicktest v1.14.4 go: downloading gomodules.xyz/envconfig v1.3.1-0.20190308184047-426f31af0d45 go: downloading gopkg.in/gomail.v2 v2.0.0-20160411212932-81ebce5c23df go: downloading github.com/gorilla/websocket v1.5.0 go: downloading github.com/go-test/deep v1.0.4 go: downloading github.com/Masterminds/goutils v1.1.1 go: downloading github.com/Masterminds/semver/v3 v3.2.0 go: downloading github.com/huandu/xstrings v1.3.3 go: downloading github.com/mitchellh/copystructure v1.2.0 go: downloading github.com/shopspring/decimal v1.2.0 go: downloading github.com/google/btree v1.1.2 go: downloading github.com/hashicorp/go-cleanhttp v0.5.2 go: downloading github.com/jpillora/backoff v1.0.0 go: downloading golang.org/x/mod v0.8.0 go: downloading github.com/kr/text v0.2.0 go: downloading github.com/onsi/ginkgo v1.14.1 go: downloading github.com/rogpeppe/go-internal v1.9.0 go: downloading github.com/Jeffail/gabs v1.4.0 go: downloading github.com/gopackage/ddp v0.0.0-20170117053602-652027933df4 go: downloading github.com/sony/sonyflake v1.0.0 go: downloading k8s.io/cluster-bootstrap v0.25.8 go: downloading github.com/docker/distribution v2.8.1+incompatible go: downloading github.com/moby/term v0.0.0-20220808134915-39b0c02b01ae go: downloading gopkg.in/alexcesaro/quotedprintable.v3 v3.0.0-20150716171945-2caba252f4dc go: downloading github.com/ProtonMail/go-crypto v0.0.0-20230217124315-7d5c6f04bbb8 go: downloading github.com/mitchellh/reflectwalk v1.0.2 go: downloading golang.org/x/sync v0.3.0 go: downloading github.com/go-errors/errors v1.4.2 go: downloading github.com/creack/pty v1.1.11 go: downloading gotest.tools/v3 v3.0.3 go: downloading github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 go: downloading github.com/monochromegane/go-gitignore v0.0.0-20200626010858-205db1a8cc00 go: downloading github.com/xlab/treeprint v1.1.0 go: downloading github.com/moby/spdystream v0.2.0 go: downloading github.com/armon/go-socks5 v0.0.0-20160902184237-e75332964ef5 go: downloading github.com/Azure/go-ansiterm v0.0.0-20210617225240-d185dfc1b5a1 go: downloading github.com/opencontainers/go-digest v1.0.0 go: downloading github.com/sergi/go-diff v1.1.0 go: downloading go.starlark.net v0.0.0-20200306205701-8dd3e2ee1dd5 go: downloading github.com/cloudflare/circl v1.3.3 go: downloading github.com/google/s2a-go v0.1.4 go: downloading go.opencensus.io v0.24.0 go: downloading github.com/googleapis/enterprise-certificate-proxy v0.2.5 go: downloading github.com/googleapis/gax-go/v2 v2.12.0 + kubectl delete ns argo-rollouts namespace "argo-rollouts" deleted + kubectl wait --timeout=5m --for=delete namespace/argo-rollouts + kubectl create ns argo-rollouts namespace/argo-rollouts created + kubectl config set-context --current --namespace=argo-rollouts Context "default/api-nas-412-ocp-gitops-qe-com:6443/kube:admin" modified. + cd /tmp/tmp.hNjWJyuQmo/argo-rollouts-manager/hack/.. + '[' -z true ']' + cd /tmp/tmp.58kGTG6WqB/argo-rollouts + cat + kubectl apply -f /tmp/tmp.58kGTG6WqB/rollout-manager.yaml rolloutmanager.argoproj.io/argo-rollout created + echo '* Waiting for Argo Rollouts Deployment to exist' * Waiting for Argo Rollouts Deployment to exist + kubectl get -n argo-rollouts deployment/argo-rollouts NAME READY UP-TO-DATE AVAILABLE AGE argo-rollouts 0/1 1 0 0s + kubectl wait --for=condition=Available --timeout=10m -n argo-rollouts deployment/argo-rollouts deployment.apps/argo-rollouts condition met + kubectl apply -f test/e2e/crds customresourcedefinition.apiextensions.k8s.io/apisixroutes.apisix.apache.org configured customresourcedefinition.apiextensions.k8s.io/wasmplugins.extensions.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/destinationrules.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/envoyfilters.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/gateways.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/proxyconfigs.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/serviceentries.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/sidecars.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/virtualservices.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/workloadentries.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/workloadgroups.networking.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/authorizationpolicies.security.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/peerauthentications.security.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/requestauthentications.security.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/telemetries.telemetry.istio.io unchanged customresourcedefinition.apiextensions.k8s.io/trafficsplits.split.smi-spec.io unchanged + oc adm policy add-scc-to-user anyuid -z argo-rollouts -n argo-rollouts clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "argo-rollouts" + oc adm policy add-scc-to-user anyuid -z default -n argo-rollouts clusterrole.rbac.authorization.k8s.io/system:openshift:scc:anyuid added: "default" + rm -f /tmp/test-e2e.log + set +e + make test-e2e + tee /tmp/test-e2e.log ./hack/installers/install-dev-tools.sh +++ dirname ./hack/installers/install-dev-tools.sh ++ cd ./hack/installers/../.. ++ pwd + PROJECT_ROOT=/tmp/tmp.58kGTG6WqB/argo-rollouts + DIST_PATH=/tmp/tmp.58kGTG6WqB/argo-rollouts/dist + PATH=/tmp/tmp.58kGTG6WqB/argo-rollouts/dist:/tmp/tmp.58kGTG6WqB/argo-rollouts/dist:/root/.nvm/versions/node/v20.12.2/bin/:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/local/go/bin + mkdir -p /tmp/tmp.58kGTG6WqB/argo-rollouts/dist + gotestsum_version=1.10.1 ++ go env GOOS + OS=linux ++ go env GOARCH + ARCH=amd64 + export TARGET_FILE=gotestsum_1.10.1_linux_amd64.tar.gz + TARGET_FILE=gotestsum_1.10.1_linux_amd64.tar.gz + temp_path=/tmp/gotestsum_1.10.1_linux_amd64.tar.gz + url=https://github.com/gotestyourself/gotestsum/releases/download/v1.10.1/gotestsum_1.10.1_linux_amd64.tar.gz + '[' -e /tmp/gotestsum_1.10.1_linux_amd64.tar.gz ']' + curl -sLf --retry 3 -o /tmp/gotestsum_1.10.1_linux_amd64.tar.gz https://github.com/gotestyourself/gotestsum/releases/download/v1.10.1/gotestsum_1.10.1_linux_amd64.tar.gz + mkdir -p /tmp/gotestsum-1.10.1 + tar -xvzf /tmp/gotestsum_1.10.1_linux_amd64.tar.gz -C /tmp/gotestsum-1.10.1 LICENSE LICENSE.md README.md gotestsum + cp /tmp/gotestsum-1.10.1/gotestsum /tmp/tmp.58kGTG6WqB/argo-rollouts/dist/gotestsum + chmod +x /tmp/tmp.58kGTG6WqB/argo-rollouts/dist/gotestsum + gotestsum --version gotestsum version 1.10.1 /tmp/tmp.58kGTG6WqB/argo-rollouts/dist/gotestsum --rerun-fails-report=rerunreport.txt --junitfile=junit.xml --format=testname --packages="./test/e2e" --rerun-fails=5 -- -timeout 60m -count 1 --tags e2e -p 1 -parallel 1 -v --short ./test/e2e PASS test/e2e.TestAnalysisSuite/TestAnalysisWithArgs (47.26s) PASS test/e2e.TestAnalysisSuite/TestAnalysisWithSecret (34.04s) PASS test/e2e.TestAnalysisSuite/TestBackgroundAnalysisWithArgs (31.52s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenAbortAndUpdate (26.24s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenAnalysis (35.31s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenKitchenSink (52.23s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenPostPromotionFail (20.79s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenPrePromotionFail (33.06s) PASS test/e2e.TestAnalysisSuite/TestCanaryBackgroundAnalysis (8.08s) PASS test/e2e.TestAnalysisSuite/TestCanaryFailInlineMultipleAnalysis (20.75s) PASS test/e2e.TestAnalysisSuite/TestCanaryInlineAnalysis (26.36s) PASS test/e2e.TestAnalysisSuite/TestCanaryInlineMultipleAnalysis (23.64s) PASS test/e2e.TestAnalysisSuite/TestMultipleAnalysis (15.51s) PASS test/e2e.TestAnalysisSuite (379.63s) === RUN TestAPISIXSuite/TestAPISIXCanarySetHeaderStep time="2024-10-30T11:30:40Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T11:30:43Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T11:30:45Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T11:30:47Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route created\n" rollout=rollout-apisix-canary time="2024-10-30T11:30:50Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T11:30:50Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T11:30:50Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (10.70s) FAIL test/e2e.TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (10.70s) PASS test/e2e.TestAPISIXSuite/TestAPISIXCanaryStep (33.42s) === RUN TestAPISIXSuite --- FAIL: TestAPISIXSuite (44.40s) FAIL test/e2e.TestAPISIXSuite (44.40s) PASS test/e2e.TestAWSSuite/TestALBExperimentStep (30.17s) PASS test/e2e.TestAWSSuite/TestALBExperimentStepMultiIngress (33.18s) PASS test/e2e.TestAWSSuite/TestALBExperimentStepNoSetWeight (24.50s) PASS test/e2e.TestAWSSuite/TestALBExperimentStepNoSetWeightMultiIngress (25.59s) PASS test/e2e.TestAWSSuite/TestALBPingPongUpdate (24.51s) PASS test/e2e.TestAWSSuite/TestALBPingPongUpdateMultiIngress (23.13s) PASS test/e2e.TestAWSSuite/TestAlbHeaderRoute (14.28s) PASS test/e2e.TestAWSSuite/TestAlbHeaderRouteMultiIngress (26.55s) PASS test/e2e.TestAWSSuite (203.65s) PASS test/e2e.TestBlueGreenSuite/TestBlueGreenPreviewReplicaCount (12.38s) PASS test/e2e.TestBlueGreenSuite/TestBlueGreenProgressDeadlineExceededWithPause (63.83s) PASS test/e2e.TestBlueGreenSuite/TestBlueGreenProgressDeadlineExceededWithoutPause (63.74s) PASS test/e2e.TestBlueGreenSuite/TestEphemeralMetadata (13.17s) PASS test/e2e.TestBlueGreenSuite (153.40s) === RUN TestCanarySuite/TestCanaryDynamicStableScale time="2024-10-30T11:37:22Z" level=info msg="Deleting e2e-test-name=TestCanaryDynamicStableScale" time="2024-10-30T11:37:24Z" level=info msg="service/dynamic-stable-scale-root created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:25Z" level=info msg="service/dynamic-stable-scale-canary created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:26Z" level=info msg="service/dynamic-stable-scale-stable created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:27Z" level=info msg="ingress.networking.k8s.io/dynamic-stable-scale-ingress created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:28Z" level=info msg="rollout.argoproj.io/dynamic-stable-scale created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:30Z" level=info msg="Marked 4 revision 1 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:30Z" level=info msg="Waiting for condition: status=Healthy" rollout=dynamic-stable-scale time="2024-10-30T11:37:31Z" level=info msg="Condition 'status=Healthy' met after 1s" rollout=dynamic-stable-scale time="2024-10-30T11:37:31Z" level=info msg="Updated rollout pod spec: 2024-10-30T11:37:31.586220824Z" rollout=dynamic-stable-scale time="2024-10-30T11:37:32Z" level=info msg="Marked 1 revision 2 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:32Z" level=info msg="Waiting for condition: status=Paused" rollout=dynamic-stable-scale time="2024-10-30T11:37:33Z" level=info msg="Condition 'status=Paused' met after 0s" rollout=dynamic-stable-scale time="2024-10-30T11:37:33Z" level=info msg="Sleeping 2s" rollout=dynamic-stable-scale time="2024-10-30T11:37:35Z" level=info msg="Pod expectation 'revision:1 pod count == 3' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:35Z" level=info msg="Pod expectation 'revision:2 pod count == 1' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:35Z" level=info msg="Promoted rollout" rollout=dynamic-stable-scale time="2024-10-30T11:37:36Z" level=info msg="Marked 2 revision 2 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:36Z" level=info msg="Waiting for condition: status.currentStepIndex=3" rollout=dynamic-stable-scale time="2024-10-30T11:37:38Z" level=info msg="Condition 'status.currentStepIndex=3' met after 1s" rollout=dynamic-stable-scale time="2024-10-30T11:37:38Z" level=info msg="Sleeping 2s" rollout=dynamic-stable-scale time="2024-10-30T11:37:40Z" level=info msg="Pod expectation 'revision:1 pod count == 1' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:40Z" level=info msg="Pod expectation 'revision:2 pod count == 3' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:40Z" level=info msg="Aborted rollout" rollout=dynamic-stable-scale time="2024-10-30T11:37:41Z" level=info msg="Marked 2 revision 1 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:41Z" level=info msg="Waiting for condition: rev=2 podcount=1" rollout=dynamic-stable-scale time="2024-10-30T11:37:43Z" level=info msg="Condition 'rev=2 podcount=1' met after 2s" rollout=dynamic-stable-scale time="2024-10-30T11:37:43Z" level=info msg="Pod expectation 'revision:1 pod count == 4' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:44Z" level=info msg="Marked 1 revision 1 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:44Z" level=info msg="Waiting for condition: rev=2 podcount=0" rollout=dynamic-stable-scale time="2024-10-30T11:37:44Z" level=info msg="Condition 'rev=2 podcount=0' met after 0s" rollout=dynamic-stable-scale canary_test.go:638: Expected dynamic-stable-scale-canary selector: map[app:dynamic-stable-scale rollouts-pod-template-hash:868d98995b]. Actual: map[app:dynamic-stable-scale rollouts-pod-template-hash:559649b568] Name: dynamic-stable-scale Namespace: argo-rollouts Status: ✖ Degraded Message: RolloutAborted: Rollout aborted update to revision 2 Strategy: Canary Step: 0/4 SetWeight: 0 ActualWeight: 0 Images: nginx:1.19-alpine (stable) Replicas: Desired: 4 Current: 4 Updated: 0 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ dynamic-stable-scale Rollout ✖ Degraded 16s ├──# revision:2 │ └──⧉ dynamic-stable-scale-559649b568 ReplicaSet • ScaledDown 13s canary │ └──□ dynamic-stable-scale-559649b568-pj5gg Pod ◌ Terminating 13s ready:1/1 └──# revision:1 └──⧉ dynamic-stable-scale-868d98995b ReplicaSet ✔ Healthy 16s stable ├──□ dynamic-stable-scale-868d98995b-sz5zh Pod ✔ Running 16s ready:1/1 ├──□ dynamic-stable-scale-868d98995b-5g8lv Pod ✔ Running 4s ready:1/1 ├──□ dynamic-stable-scale-868d98995b-b2wmr Pod ✔ Running 4s ready:1/1 └──□ dynamic-stable-scale-868d98995b-n4zsx Pod ✔ Running 4s ready:1/1 --- apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: annotations: rollout.argoproj.io/revision: "2" creationTimestamp: "2024-10-30T11:37:28Z" generation: 2 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestCanaryDynamicStableScale name: dynamic-stable-scale namespace: argo-rollouts resourceVersion: "20301797" uid: c0a14511-b4c0-4f42-a0bc-471fe70d235d spec: replicas: 4 selector: matchLabels: app: dynamic-stable-scale strategy: canary: canaryService: dynamic-stable-scale-canary dynamicStableScale: true stableService: dynamic-stable-scale-stable steps: - setWeight: 25 - pause: {} - setWeight: 75 - pause: {} trafficRouting: alb: ingress: dynamic-stable-scale-ingress rootService: dynamic-stable-scale-root servicePort: 80 template: metadata: annotations: update: "2024-10-30T11:37:31.586220824Z" creationTimestamp: null labels: app: dynamic-stable-scale spec: containers: - image: nginx:1.19-alpine name: dynamic-stable-scale ports: - containerPort: 80 name: http protocol: TCP resources: requests: cpu: 5m memory: 16Mi readinessGates: - conditionType: argoproj.io/e2e-readiness status: HPAReplicas: 4 abort: true abortedAt: "2024-10-30T11:37:40Z" availableReplicas: 4 blueGreen: {} canary: weights: canary: podTemplateHash: 559649b568 serviceName: dynamic-stable-scale-canary weight: 0 stable: podTemplateHash: 868d98995b serviceName: dynamic-stable-scale-stable weight: 100 conditions: - lastTransitionTime: "2024-10-30T11:37:31Z" lastUpdateTime: "2024-10-30T11:37:31Z" message: Rollout has minimum availability reason: AvailableReason status: "True" type: Available - lastTransitionTime: "2024-10-30T11:37:31Z" lastUpdateTime: "2024-10-30T11:37:31Z" message: Rollout is not healthy reason: RolloutHealthy status: "False" type: Healthy - lastTransitionTime: "2024-10-30T11:37:31Z" lastUpdateTime: "2024-10-30T11:37:31Z" message: RolloutCompleted reason: RolloutCompleted status: "False" type: Completed - lastTransitionTime: "2024-10-30T11:37:38Z" lastUpdateTime: "2024-10-30T11:37:38Z" message: Rollout is paused reason: RolloutPaused status: "True" type: Paused - lastTransitionTime: "2024-10-30T11:37:40Z" lastUpdateTime: "2024-10-30T11:37:40Z" message: Rollout aborted update to revision 2 reason: RolloutAborted status: "False" type: Progressing currentPodHash: 559649b568 currentStepHash: 7c5b668566 currentStepIndex: 0 message: 'RolloutAborted: Rollout aborted update to revision 2' observedGeneration: "2" phase: Degraded readyReplicas: 4 replicas: 4 selector: app=dynamic-stable-scale stableRS: 868d98995b 2024-10-30T11:37:28Z Normal RolloutAddedToInformer Rollout resource added to informer: argo-rollouts/dynamic-stable-scale 2024-10-30T11:37:28Z Normal RolloutNotCompleted Rollout not completed, started update to revision 1 (868d98995b) 2024-10-30T11:37:28Z Normal RolloutUpdated Rollout updated to revision 1 2024-10-30T11:37:28Z Normal NewReplicaSetCreated Created ReplicaSet dynamic-stable-scale-868d98995b (revision 1) 2024-10-30T11:37:28Z Normal PatchingALBIngress Updating Ingress `dynamic-stable-scale-ingress` to desiredWeight '0' 2024-10-30T11:37:28Z Normal TrafficWeightUpdated Traffic weight updated to 0 2024-10-30T11:37:28Z Normal ScalingReplicaSet Scaled up ReplicaSet dynamic-stable-scale-868d98995b (revision 1) from 0 to 4 2024-10-30T11:37:28Z Normal RolloutCompleted Rollout completed update to revision 1 (868d98995b): Initial deploy 2024-10-30T11:37:31Z (x2) Normal TrafficWeightUpdated Traffic weight updated 2024-10-30T11:37:31Z Normal SwitchService Switched selector for service 'dynamic-stable-scale-stable' from '' to '868d98995b' 2024-10-30T11:37:31Z Normal SwitchService Switched selector for service 'dynamic-stable-scale-canary' from '' to '868d98995b' 2024-10-30T11:37:31Z Normal RolloutNotCompleted Rollout not completed, started update to revision 2 (559649b568) 2024-10-30T11:37:31Z Normal RolloutUpdated Rollout updated to revision 2 2024-10-30T11:37:31Z Normal NewReplicaSetCreated Created ReplicaSet dynamic-stable-scale-559649b568 (revision 2) 2024-10-30T11:37:31Z Normal ScalingReplicaSet Scaled up ReplicaSet dynamic-stable-scale-559649b568 (revision 2) from 0 to 1 2024-10-30T11:37:33Z Normal SwitchService Switched selector for service 'dynamic-stable-scale-canary' from '868d98995b' to '559649b568' 2024-10-30T11:37:33Z Normal PatchingALBIngress Updating Ingress `dynamic-stable-scale-ingress` to desiredWeight '25' 2024-10-30T11:37:33Z Normal TrafficWeightUpdated Traffic weight updated from 0 to 25 2024-10-30T11:37:33Z Normal RolloutStepCompleted Rollout step 1/4 completed (setWeight: 25) 2024-10-30T11:37:33Z Normal ScalingReplicaSet Scaled down ReplicaSet dynamic-stable-scale-868d98995b (revision 1) from 4 to 3 2024-10-30T11:37:33Z Normal RolloutPaused Rollout is paused (CanaryPauseStep) 2024-10-30T11:37:35Z Normal RolloutStepCompleted Rollout step 2/4 completed (pause) 2024-10-30T11:37:35Z Normal ScalingReplicaSet Scaled up ReplicaSet dynamic-stable-scale-559649b568 (revision 2) from 1 to 3 2024-10-30T11:37:38Z Normal PatchingALBIngress Updating Ingress `dynamic-stable-scale-ingress` to desiredWeight '75' time="2024-10-30T11:37:44Z" level=info msg="Deleting e2e-test-name=TestCanaryDynamicStableScale" --- FAIL: TestCanarySuite/TestCanaryDynamicStableScale (22.38s) FAIL test/e2e.TestCanarySuite/TestCanaryDynamicStableScale (22.38s) PASS test/e2e.TestCanarySuite/TestCanaryDynamicStableScaleRollbackToStable (18.35s) PASS test/e2e.TestCanarySuite/TestCanaryProgressDeadlineExceededWithPause (123.59s) PASS test/e2e.TestCanarySuite/TestCanaryScaleDownDelay (19.32s) === RUN TestCanarySuite/TestCanaryScaleDownOnAbort time="2024-10-30T11:40:26Z" level=info msg="Deleting e2e-test-name=TestCanaryScaleDownOnAbort" time="2024-10-30T11:40:27Z" level=info msg="service/canary-scaledowndelay-root created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:29Z" level=info msg="service/canary-scaledowndelay-canary created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:30Z" level=info msg="service/canary-scaledowndelay-stable created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:31Z" level=info msg="ingress.networking.k8s.io/canary-scaledowndelay-ingress created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:32Z" level=info msg="rollout.argoproj.io/canary-scaledownd-on-abort created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:32Z" level=info msg="Waiting for condition: status=Healthy" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:35Z" level=info msg="Condition 'status=Healthy' met after 3s" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:35Z" level=info msg="Updated rollout pod spec: 2024-10-30T11:40:35.782839303Z" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:35Z" level=info msg="Waiting for condition: status=Paused" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:38Z" level=info msg="Condition 'status=Paused' met after 2s" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:38Z" level=info msg="Aborted rollout" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:38Z" level=info msg="Waiting for condition: status=Degraded" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:38Z" level=info msg="Condition 'status=Degraded' met after 0s" rollout=canary-scaledownd-on-abort canary_test.go:544: Expected canary-scaledowndelay-canary selector: map[app:canary-scaledowndelay rollouts-pod-template-hash:66597877b7]. Actual: map[app:canary-scaledowndelay rollouts-pod-template-hash:5987d8856f] Name: canary-scaledownd-on-abort Namespace: argo-rollouts Status: ✖ Degraded Message: RolloutAborted: Rollout aborted update to revision 2 Strategy: Canary Step: 0/2 SetWeight: 0 ActualWeight: 0 Images: nginx:1.19-alpine (canary, stable) Replicas: Desired: 1 Current: 2 Updated: 1 Ready: 2 Available: 2 NAME KIND STATUS AGE INFO ⟳ canary-scaledownd-on-abort Rollout ✖ Degraded 6s ├──# revision:2 │ └──⧉ canary-scaledownd-on-abort-5987d8856f ReplicaSet ✔ Healthy 3s canary,delay:0s │ └──□ canary-scaledownd-on-abort-5987d8856f-vrh9p Pod ✔ Running 3s ready:1/1 └──# revision:1 └──⧉ canary-scaledownd-on-abort-66597877b7 ReplicaSet ✔ Healthy 6s stable └──□ canary-scaledownd-on-abort-66597877b7-qz8sp Pod ✔ Running 6s ready:1/1 --- apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: annotations: rollout.argoproj.io/revision: "2" creationTimestamp: "2024-10-30T11:40:32Z" generation: 3 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestCanaryScaleDownOnAbort name: canary-scaledownd-on-abort namespace: argo-rollouts resourceVersion: "20306031" uid: bcc0b471-85ba-4c60-8b65-032d25b865b9 spec: replicas: 1 selector: matchLabels: app: canary-scaledowndelay strategy: canary: abortScaleDownDelaySeconds: 1 canaryService: canary-scaledowndelay-canary scaleDownDelayRevisionLimit: 1 stableService: canary-scaledowndelay-stable steps: - setCanaryScale: replicas: 1 - pause: {} trafficRouting: alb: ingress: canary-scaledowndelay-ingress rootService: canary-scaledowndelay-root servicePort: 80 template: metadata: annotations: update: "2024-10-30T11:40:35.782839303Z" creationTimestamp: null labels: app: canary-scaledowndelay spec: containers: - image: nginx:1.19-alpine name: canary-scaledowndelay ports: - containerPort: 80 name: http protocol: TCP resources: requests: cpu: 5m memory: 16Mi status: HPAReplicas: 2 abort: true abortedAt: "2024-10-30T11:40:38Z" availableReplicas: 2 blueGreen: {} canary: weights: canary: podTemplateHash: 5987d8856f serviceName: canary-scaledowndelay-canary weight: 0 stable: podTemplateHash: 66597877b7 serviceName: canary-scaledowndelay-stable weight: 100 conditions: - lastTransitionTime: "2024-10-30T11:40:35Z" lastUpdateTime: "2024-10-30T11:40:35Z" message: Rollout has minimum availability reason: AvailableReason status: "True" type: Available - lastTransitionTime: "2024-10-30T11:40:35Z" lastUpdateTime: "2024-10-30T11:40:35Z" message: Rollout is not healthy reason: RolloutHealthy status: "False" type: Healthy - lastTransitionTime: "2024-10-30T11:40:35Z" lastUpdateTime: "2024-10-30T11:40:35Z" message: RolloutCompleted reason: RolloutCompleted status: "False" type: Completed - lastTransitionTime: "2024-10-30T11:40:38Z" lastUpdateTime: "2024-10-30T11:40:38Z" message: Rollout is paused reason: RolloutPaused status: "True" type: Paused - lastTransitionTime: "2024-10-30T11:40:38Z" lastUpdateTime: "2024-10-30T11:40:38Z" message: Rollout aborted update to revision 2 reason: RolloutAborted status: "False" type: Progressing currentPodHash: 5987d8856f currentStepHash: 5db8f9b897 currentStepIndex: 0 message: 'RolloutAborted: Rollout aborted update to revision 2' observedGeneration: "3" phase: Degraded readyReplicas: 2 replicas: 2 selector: app=canary-scaledowndelay stableRS: 66597877b7 updatedReplicas: 1 2024-10-30T11:40:32Z Normal RolloutAddedToInformer Rollout resource added to informer: argo-rollouts/canary-scaledownd-on-abort 2024-10-30T11:40:32Z Normal RolloutNotCompleted Rollout not completed, started update to revision 1 (66597877b7) 2024-10-30T11:40:32Z Normal RolloutUpdated Rollout updated to revision 1 2024-10-30T11:40:32Z Normal NewReplicaSetCreated Created ReplicaSet canary-scaledownd-on-abort-66597877b7 (revision 1) 2024-10-30T11:40:32Z Normal PatchingALBIngress Updating Ingress `canary-scaledowndelay-ingress` to desiredWeight '0' 2024-10-30T11:40:32Z Normal TrafficWeightUpdated Traffic weight updated to 0 2024-10-30T11:40:32Z Normal ScalingReplicaSet Scaled up ReplicaSet canary-scaledownd-on-abort-66597877b7 (revision 1) from 0 to 1 2024-10-30T11:40:32Z Normal RolloutCompleted Rollout completed update to revision 1 (66597877b7): Initial deploy 2024-10-30T11:40:35Z (x2) Normal TrafficWeightUpdated Traffic weight updated 2024-10-30T11:40:35Z Normal SwitchService Switched selector for service 'canary-scaledowndelay-stable' from '' to '66597877b7' 2024-10-30T11:40:35Z Normal SwitchService Switched selector for service 'canary-scaledowndelay-canary' from '' to '66597877b7' 2024-10-30T11:40:35Z Normal RolloutNotCompleted Rollout not completed, started update to revision 2 (5987d8856f) 2024-10-30T11:40:35Z Normal RolloutUpdated Rollout updated to revision 2 2024-10-30T11:40:35Z Normal NewReplicaSetCreated Created ReplicaSet canary-scaledownd-on-abort-5987d8856f (revision 2) 2024-10-30T11:40:35Z Normal ScalingReplicaSet Scaled up ReplicaSet canary-scaledownd-on-abort-5987d8856f (revision 2) from 0 to 1 2024-10-30T11:40:38Z (x3) Normal SwitchService Switched selector for service 'canary-scaledowndelay-canary' from '66597877b7' to '5987d8856f' 2024-10-30T11:40:38Z Normal RolloutStepCompleted Rollout step 1/2 completed (setCanaryScale{replicas: 1}) 2024-10-30T11:40:38Z Normal RolloutPaused Rollout is paused (CanaryPauseStep) 2024-10-30T11:40:38Z (x3) Normal SwitchService Switched selector for service 'canary-scaledowndelay-canary' from '5987d8856f' to '66597877b7' 2024-10-30T11:40:38Z Warning RolloutAborted Rollout aborted update to revision 2 time="2024-10-30T11:40:38Z" level=info msg="Deleting e2e-test-name=TestCanaryScaleDownOnAbort" --- FAIL: TestCanarySuite/TestCanaryScaleDownOnAbort (12.73s) FAIL test/e2e.TestCanarySuite/TestCanaryScaleDownOnAbort (12.73s) PASS test/e2e.TestCanarySuite/TestCanaryScaleDownOnAbortNoTrafficRouting (14.88s) PASS test/e2e.TestCanarySuite/TestCanaryUnScaleDownOnAbort (13.83s) PASS test/e2e.TestCanarySuite/TestCanaryWithPausedRollout (101.58s) PASS test/e2e.TestCanarySuite/TestEphemeralMetadata (12.58s) PASS test/e2e.TestCanarySuite/TestReduceWeightAndHonorMaxUnavailable (35.13s) PASS test/e2e.TestCanarySuite/TestRolloutScalingDuringUpdate (5.34s) PASS test/e2e.TestCanarySuite/TestRolloutScalingWhenPaused (10.75s) PASS test/e2e.TestCanarySuite/TestRolloutWithMaxSurgeScalingDuringUpdate (5.42s) PASS test/e2e.TestCanarySuite/TestCanarySetCanaryScale (17.75s) === RUN TestCanarySuite time="2024-10-30T11:37:22Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" --- FAIL: TestCanarySuite (397.49s) FAIL test/e2e.TestCanarySuite (397.49s) === RUN TestExperimentSuite/TestExperimentWithDryRunMetrics time="2024-10-30T11:44:16Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T11:44:18Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T11:44:19Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T11:44:19Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T11:44:52Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T11:44:52Z" level=info msg="Sleeping 3s" time="2024-10-30T11:44:55Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T11:44:55Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T11:44:55Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-n6tsk Pod ✔ Running 36s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ fcf03914-e255-4633-bb07-15086c587af4.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T11:44:19Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20313600" uid: fa3b5e30-cc72-4d4d-aee8-8e4bf4f86d9e spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T11:44:22Z" conditions: - lastTransitionTime: "2024-10-30T11:44:52Z" lastUpdateTime: "2024-10-30T11:44:52Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T11:44:52Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T11:44:19Z (x2) Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T11:44:19Z (x2) Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T11:44:19Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T11:44:22Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T11:44:22Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T11:44:22Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T11:44:22Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T11:44:52Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T11:44:52Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T11:44:52Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T11:44:55Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.60s) FAIL test/e2e.TestExperimentSuite/TestExperimentWithDryRunMetrics (38.60s) PASS test/e2e.TestExperimentSuite/TestExperimentWithMeasurementRetentionMetrics (37.85s) PASS test/e2e.TestExperimentSuite/TestExperimentWithMultiportServiceAndScaleDownDelay (19.06s) PASS test/e2e.TestExperimentSuite/TestExperimentWithServiceAndScaleDownDelay (19.07s) PASS test/e2e.TestExperimentSuite/TestExperimentWithServiceNameAndScaleDownDelay (18.97s) PASS test/e2e.TestExperimentSuite/TestRolloutWithExperimentAndAnalysis (16.56s) === RUN TestExperimentSuite --- FAIL: TestExperimentSuite (134.06s) FAIL test/e2e.TestExperimentSuite (134.06s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenExceedProgressDeadlineAbort (12.04s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenPreviewReplicaCountPromoteFull (12.85s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenPromoteFull (8.94s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenScaleDownDelay (17.40s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenScaleDownOnAbort (12.95s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenToCanary (9.04s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenUpdate (9.37s) PASS test/e2e.TestFunctionalSuite/TestCanaryPromoteFull (11.96s) PASS test/e2e.TestFunctionalSuite/TestCompleteRolloutRestart (23.68s) PASS test/e2e.TestFunctionalSuite/TestContainerResourceFormats (4.83s) === RUN TestFunctionalSuite/TestControllerMetrics time="2024-10-30T11:48:51Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T11:48:51Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 6147 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc0017e9d40, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc000f69000) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc001529100?, 0xc0009327f8?, 0xc001104400?}, {0x3212d41, 0x4}, {0xc001601e70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc001529100?, 0xc0009327f8?, 0xc000f69000?}, {0xc001601e70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc0017e9d40) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc0017e9d40, 0xc000806870) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 5156 /usr/local/go/src/testing/testing.go:1648 +0x3ad --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.49s) FAIL test/e2e.TestFunctionalSuite/TestControllerMetrics (0.49s) PASS test/e2e.TestFunctionalSuite/TestFixInvalidSpec (21.62s) PASS test/e2e.TestFunctionalSuite/TestKubectlWaitForCompleted (23.50s) PASS test/e2e.TestFunctionalSuite/TestKubectlWaitForPaused (8.61s) PASS test/e2e.TestFunctionalSuite/TestMalformedRollout (5.10s) PASS test/e2e.TestFunctionalSuite/TestMalformedRolloutEphemeralCtr (4.85s) === RUN TestFunctionalSuite/TestRolloutAbortRetryPromote time="2024-10-30T11:49:55Z" level=info msg="Deleting e2e-test-name=TestRolloutAbortRetryPromote" time="2024-10-30T11:49:55Z" level=info msg="Event watcher started" time="2024-10-30T11:49:57Z" level=info msg="rollout.argoproj.io/abort-retry-promote created\n" rollout=abort-retry-promote time="2024-10-30T11:49:57Z" level=info msg="Waiting for condition: status=Healthy" rollout=abort-retry-promote time="2024-10-30T11:49:59Z" level=info msg="Condition 'status=Healthy' met after 2s" rollout=abort-retry-promote time="2024-10-30T11:49:59Z" level=info msg="Updated rollout pod spec: 2024-10-30T11:49:59.915577022Z" rollout=abort-retry-promote time="2024-10-30T11:49:59Z" level=info msg="Waiting for condition: status=Paused" rollout=abort-retry-promote when.go:455: timeout after 2m0s waiting for condition status=Paused time="2024-10-30T11:51:59Z" level=info msg="Event watcher stopped" rollout=abort-retry-promote Name: abort-retry-promote Namespace: argo-rollouts Status: ◌ Progressing Message: waiting for rollout spec update to be observed Strategy: Canary Step: 2/2 SetWeight: 100 ActualWeight: 100 Images: nginx:1.19-alpine (stable) Replicas: Desired: 1 Current: 1 Updated: 1 Ready: 1 Available: 1 NAME KIND STATUS AGE INFO ⟳ abort-retry-promote Rollout ◌ Progressing 2m3s └──# revision:1 └──⧉ abort-retry-promote-7fd9b5545c ReplicaSet ✔ Healthy 2m3s stable └──□ abort-retry-promote-7fd9b5545c-v79v7 Pod ✔ Running 2m3s ready:1/1 --- apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: annotations: rollout.argoproj.io/revision: "1" creationTimestamp: "2024-10-30T11:49:57Z" generation: 3 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestRolloutAbortRetryPromote name: abort-retry-promote namespace: argo-rollouts resourceVersion: "20321690" uid: 97ddbad8-8c88-4850-ada1-289d2c5dc02c spec: replicas: 1 selector: matchLabels: app: abort-retry-promote strategy: canary: steps: - setWeight: 50 - pause: duration: 3s template: metadata: annotations: update: "2024-10-30T11:49:59.915577022Z" creationTimestamp: null labels: app: abort-retry-promote spec: containers: - image: nginx:1.19-alpine name: abort-retry-promote resources: requests: cpu: 1m memory: 16Mi status: HPAReplicas: 1 availableReplicas: 1 blueGreen: {} canary: {} conditions: - lastTransitionTime: "2024-10-30T11:49:57Z" lastUpdateTime: "2024-10-30T11:49:57Z" message: RolloutCompleted reason: RolloutCompleted status: "True" type: Completed - lastTransitionTime: "2024-10-30T11:49:59Z" lastUpdateTime: "2024-10-30T11:49:59Z" message: Rollout is healthy reason: RolloutHealthy status: "True" type: Healthy - lastTransitionTime: "2024-10-30T11:49:57Z" lastUpdateTime: "2024-10-30T11:49:59Z" message: ReplicaSet "abort-retry-promote-7fd9b5545c" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing - lastTransitionTime: "2024-10-30T11:49:59Z" lastUpdateTime: "2024-10-30T11:49:59Z" message: Rollout has minimum availability reason: AvailableReason status: "True" type: Available currentPodHash: 7fd9b5545c currentStepHash: 77fbc56786 currentStepIndex: 2 observedGeneration: "2" phase: Healthy readyReplicas: 1 replicas: 1 selector: app=abort-retry-promote stableRS: 7fd9b5545c updatedReplicas: 1 2024-10-30T11:49:57Z Normal RolloutAddedToInformer Rollout resource added to informer: argo-rollouts/abort-retry-promote 2024-10-30T11:49:57Z Normal RolloutNotCompleted Rollout not completed, started update to revision 1 (7fd9b5545c) 2024-10-30T11:49:57Z Normal RolloutUpdated Rollout updated to revision 1 2024-10-30T11:49:57Z Normal NewReplicaSetCreated Created ReplicaSet abort-retry-promote-7fd9b5545c (revision 1) 2024-10-30T11:49:57Z Normal ScalingReplicaSet Scaled up ReplicaSet abort-retry-promote-7fd9b5545c (revision 1) from 0 to 1 2024-10-30T11:49:57Z Normal RolloutCompleted Rollout completed update to revision 1 (7fd9b5545c): Initial deploy time="2024-10-30T11:52:00Z" level=info msg="Deleting e2e-test-name=TestRolloutAbortRetryPromote" --- FAIL: TestFunctionalSuite/TestRolloutAbortRetryPromote (124.78s) FAIL test/e2e.TestFunctionalSuite/TestRolloutAbortRetryPromote (124.78s) PASS test/e2e.TestFunctionalSuite/TestRolloutBadToBadUpdate (22.94s) PASS test/e2e.TestFunctionalSuite/TestRolloutBadToGoodSetWeightUpdate (26.92s) PASS test/e2e.TestFunctionalSuite/TestRolloutBadToGoodUpdate (24.18s) PASS test/e2e.TestFunctionalSuite/TestRolloutGoodToBadUpdate (26.93s) PASS test/e2e.TestFunctionalSuite/TestRolloutPDBRestart (49.84s) PASS test/e2e.TestFunctionalSuite/TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds (41.98s) PASS test/e2e.TestFunctionalSuite/TestRolloutPodVolumesItemsSpec (10.60s) PASS test/e2e.TestFunctionalSuite/TestRolloutRestart (21.15s) PASS test/e2e.TestFunctionalSuite/TestWorkloadRef (19.26s) PASS test/e2e.TestFunctionalSuite/TestWorkloadRefTemplate (2.78s) === RUN TestFunctionalSuite time="2024-10-30T11:46:48Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" --- FAIL: TestFunctionalSuite (560.01s) FAIL test/e2e.TestFunctionalSuite (560.01s) PASS test/e2e.TestHeaderRoutingSuite/TestIstioHostHeaderRoute (19.83s) PASS test/e2e.TestHeaderRoutingSuite (20.12s) PASS test/e2e.TestIstioSuite/TestIstioAbortUpdate (10.96s) PASS test/e2e.TestIstioSuite/TestIstioAbortUpdateDeleteAllCanaryPods (14.88s) PASS test/e2e.TestIstioSuite/TestIstioHostSplit (23.54s) PASS test/e2e.TestIstioSuite/TestIstioHostSplitExperimentStep (21.02s) PASS test/e2e.TestIstioSuite/TestIstioHostSplitOnlyTls (12.10s) PASS test/e2e.TestIstioSuite/TestIstioSubsetSplit (13.80s) PASS test/e2e.TestIstioSuite/TestIstioSubsetSplitExperimentStep (22.23s) PASS test/e2e.TestIstioSuite/TestIstioSubsetSplitSingleRoute (11.58s) PASS test/e2e.TestIstioSuite/TestIstioUpdateInMiddleZeroCanaryReplicas (12.91s) PASS test/e2e.TestIstioSuite (143.30s) PASS test/e2e.TestMirrorRouteSuite/TestIstioHostMirrorRoute (21.29s) PASS test/e2e.TestMirrorRouteSuite (21.57s) PASS test/e2e.TestRollbackSuite/TestRollbackAnalysisOutsideWindow (57.30s) PASS test/e2e.TestRollbackSuite/TestRollbackAnalysisWithinWindow (44.77s) PASS test/e2e.TestRollbackSuite (104.77s) FAIL test/e2e DONE 104 tests, 6 skipped, 10 failures in 2799.712s === RUN TestAPISIXSuite/TestAPISIXCanarySetHeaderStep time="2024-10-30T12:01:07Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:01:09Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:01:10Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:01:11Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route configured\n" rollout=rollout-apisix-canary time="2024-10-30T12:01:12Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:01:12Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:01:12Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:01:12Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (5.16s) FAIL test/e2e.TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 1) (5.16s) === RUN TestAPISIXSuite --- FAIL: TestAPISIXSuite (5.53s) FAIL test/e2e.TestAPISIXSuite (re-run 1) (5.53s) FAIL test/e2e PASS test/e2e.TestCanarySuite/TestCanaryDynamicStableScale (re-run 1) (23.71s) PASS test/e2e.TestCanarySuite (re-run 1) (25.22s) PASS test/e2e PASS test/e2e.TestCanarySuite/TestCanaryScaleDownOnAbort (re-run 1) (14.58s) PASS test/e2e.TestCanarySuite (re-run 1) (16.21s) PASS test/e2e === RUN TestExperimentSuite/TestExperimentWithDryRunMetrics time="2024-10-30T12:02:24Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:02:26Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:02:27Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:02:27Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:03:00Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T12:03:00Z" level=info msg="Sleeping 3s" time="2024-10-30T12:03:03Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:03:03Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:03:03Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-pv5sn Pod ✔ Running 36s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ ef43dcb9-fef6-45df-8526-f52e1699df8e.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:02:27Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20342675" uid: ed9953aa-fcab-4976-bb6c-b5dc2ab5d7cd spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:02:30Z" conditions: - lastTransitionTime: "2024-10-30T12:03:00Z" lastUpdateTime: "2024-10-30T12:03:00Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:03:00Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:02:27Z Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:02:27Z Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:02:27Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:02:30Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:02:30Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:02:30Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:02:30Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:03:00Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:03:00Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:03:00Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:03:03Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:03:03Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.89s) FAIL test/e2e.TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 1) (38.89s) === RUN TestExperimentSuite --- FAIL: TestExperimentSuite (39.23s) FAIL test/e2e.TestExperimentSuite (re-run 1) (39.23s) FAIL test/e2e === RUN TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:03:14Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:03:14Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 123 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc0001a7ba0, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc000c3c000) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc00055fa40?, 0xc0004d8270?, 0xc000759ca0?}, {0x3212d41, 0x4}, {0xc0007ffe70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc00055fa40?, 0xc0004d8270?, 0xc000c3c000?}, {0xc0007ffe70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc0001a7ba0) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc0001a7ba0, 0xc000563680) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 82 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:03:15Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.57s) FAIL test/e2e.TestFunctionalSuite/TestControllerMetrics (re-run 1) (0.57s) === RUN TestFunctionalSuite time="2024-10-30T12:03:14Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" --- FAIL: TestFunctionalSuite (2.26s) FAIL test/e2e.TestFunctionalSuite (re-run 1) (2.26s) FAIL test/e2e PASS test/e2e.TestFunctionalSuite/TestRolloutAbortRetryPromote (re-run 1) (13.95s) PASS test/e2e.TestFunctionalSuite (re-run 1) (15.45s) PASS test/e2e DONE 2 runs, 116 tests, 6 skipped, 16 failures in 2963.783s === RUN TestAPISIXSuite/TestAPISIXCanarySetHeaderStep time="2024-10-30T12:03:51Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:03:52Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:03:53Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:03:54Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:03:55Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:03:55Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:03:55Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:03:56Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (5.08s) FAIL test/e2e.TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 2) (5.08s) === RUN TestAPISIXSuite --- FAIL: TestAPISIXSuite (5.42s) FAIL test/e2e.TestAPISIXSuite (re-run 2) (5.42s) FAIL test/e2e === RUN TestExperimentSuite/TestExperimentWithDryRunMetrics time="2024-10-30T12:04:06Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:04:07Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:04:08Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:04:08Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:04:41Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T12:04:41Z" level=info msg="Sleeping 3s" time="2024-10-30T12:04:44Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:04:44Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:04:44Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-lrkmk Pod ✔ Running 36s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ 2909fff8-b8da-42f3-b43b-c3965df56dac.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:04:08Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20344918" uid: e4adc1cb-2c41-47fc-85c1-845b82ab7368 spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:04:11Z" conditions: - lastTransitionTime: "2024-10-30T12:04:41Z" lastUpdateTime: "2024-10-30T12:04:41Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:04:41Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:04:08Z Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:04:08Z Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:04:08Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:04:11Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:04:11Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:04:11Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:04:11Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:04:41Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:04:41Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:04:41Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:04:44Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:04:44Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.34s) FAIL test/e2e.TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 2) (38.34s) === RUN TestExperimentSuite --- FAIL: TestExperimentSuite (38.68s) FAIL test/e2e.TestExperimentSuite (re-run 2) (38.68s) FAIL test/e2e === RUN TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:04:56Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:04:56Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 124 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc0001a6820, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc0001ccb00) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc000a93340?, 0xc0009ce348?, 0xc0009f7910?}, {0x3212d41, 0x4}, {0xc000845e70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc000a93340?, 0xc0009ce348?, 0xc0001ccb00?}, {0xc000845e70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc0001a6820) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc0001a6820, 0xc000847cb0) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 15 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:04:57Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.57s) FAIL test/e2e.TestFunctionalSuite/TestControllerMetrics (re-run 2) (0.57s) === RUN TestFunctionalSuite time="2024-10-30T12:04:56Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" --- FAIL: TestFunctionalSuite (2.09s) FAIL test/e2e.TestFunctionalSuite (re-run 2) (2.09s) FAIL test/e2e DONE 3 runs, 122 tests, 6 skipped, 22 failures in 3039.412s === RUN TestAPISIXSuite/TestAPISIXCanarySetHeaderStep time="2024-10-30T12:05:07Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:05:08Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:05:09Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:05:10Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:05:11Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:05:11Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:05:11Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:05:12Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.99s) FAIL test/e2e.TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 3) (4.99s) === RUN TestAPISIXSuite --- FAIL: TestAPISIXSuite (5.34s) FAIL test/e2e.TestAPISIXSuite (re-run 3) (5.34s) FAIL test/e2e === RUN TestExperimentSuite/TestExperimentWithDryRunMetrics time="2024-10-30T12:05:21Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:05:23Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:05:24Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:05:24Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:05:57Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T12:05:57Z" level=info msg="Sleeping 3s" time="2024-10-30T12:06:00Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:06:00Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:06:00Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-x227n Pod ✔ Running 35s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ 68e50f3e-5c93-4e93-b1df-894005693883.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:05:24Z" generation: 10 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20346544" uid: fef5a489-2501-4c18-877b-a588e257b1e2 spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:05:27Z" conditions: - lastTransitionTime: "2024-10-30T12:05:57Z" lastUpdateTime: "2024-10-30T12:05:57Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:05:57Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:05:24Z Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:05:24Z Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:05:25Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:05:27Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:05:27Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:05:27Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:05:27Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:05:57Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:05:57Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:05:57Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:06:00Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:06:00Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.59s) FAIL test/e2e.TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 3) (38.59s) === RUN TestExperimentSuite --- FAIL: TestExperimentSuite (38.95s) FAIL test/e2e.TestExperimentSuite (re-run 3) (38.95s) FAIL test/e2e === RUN TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:06:11Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:06:11Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 78 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc000102b60, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc00050e200) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc00039bd80?, 0xc000a028a8?, 0xc0009bc410?}, {0x3212d41, 0x4}, {0xc000973e70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc00039bd80?, 0xc000a028a8?, 0xc00050e200?}, {0xc000973e70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc000102b60) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc000102b60, 0xc0002fba70) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 69 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:06:11Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.54s) FAIL test/e2e.TestFunctionalSuite/TestControllerMetrics (re-run 3) (0.54s) === RUN TestFunctionalSuite time="2024-10-30T12:06:11Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" --- FAIL: TestFunctionalSuite (1.95s) FAIL test/e2e.TestFunctionalSuite (re-run 3) (1.95s) FAIL test/e2e DONE 4 runs, 128 tests, 6 skipped, 28 failures in 3114.179s === RUN TestAPISIXSuite/TestAPISIXCanarySetHeaderStep time="2024-10-30T12:06:21Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:06:22Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:06:23Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:06:24Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:06:25Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:06:25Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:06:25Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:06:26Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.79s) FAIL test/e2e.TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 4) (4.79s) === RUN TestAPISIXSuite --- FAIL: TestAPISIXSuite (5.13s) FAIL test/e2e.TestAPISIXSuite (re-run 4) (5.13s) FAIL test/e2e === RUN TestExperimentSuite/TestExperimentWithDryRunMetrics time="2024-10-30T12:06:35Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:06:36Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:06:38Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:06:38Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:07:10Z" level=info msg="Condition 'phase=Successful' met after 31s" time="2024-10-30T12:07:10Z" level=info msg="Sleeping 3s" time="2024-10-30T12:07:13Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:07:13Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:07:13Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 35s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 35s delay:26s │ └──□ experiment-with-dry-run-baseline-sp272 Pod ✔ Running 35s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ fb2207c3-76b3-4e8a-9a15-d3c8e5d5d2a4.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:06:38Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20348070" uid: 558e8113-13af-4088-a763-0b0f30a834e1 spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:06:40Z" conditions: - lastTransitionTime: "2024-10-30T12:07:10Z" lastUpdateTime: "2024-10-30T12:07:10Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:07:10Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:06:38Z (x2) Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:06:38Z (x2) Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:06:38Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:06:40Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:06:40Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:06:40Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:07:08Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:07:10Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:07:10Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:07:10Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:07:13Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:07:13Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (37.82s) FAIL test/e2e.TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 4) (37.82s) === RUN TestExperimentSuite --- FAIL: TestExperimentSuite (38.17s) FAIL test/e2e.TestExperimentSuite (re-run 4) (38.17s) FAIL test/e2e === RUN TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:07:24Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:07:24Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 73 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc000c02340, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc0005f4d00) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc0005f2840?, 0xc000816160?, 0xc0002f25a0?}, {0x3212d41, 0x4}, {0xc00085de70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc0005f2840?, 0xc000816160?, 0xc0005f4d00?}, {0xc00085de70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc000c02340) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc000c02340, 0xc000b165a0) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 14 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:07:24Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.67s) FAIL test/e2e.TestFunctionalSuite/TestControllerMetrics (re-run 4) (0.67s) === RUN TestFunctionalSuite time="2024-10-30T12:07:24Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" --- FAIL: TestFunctionalSuite (2.20s) FAIL test/e2e.TestFunctionalSuite (re-run 4) (2.20s) FAIL test/e2e DONE 5 runs, 134 tests, 6 skipped, 34 failures in 3187.316s === RUN TestAPISIXSuite/TestAPISIXCanarySetHeaderStep time="2024-10-30T12:07:34Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:07:35Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:07:36Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:07:37Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:07:38Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:07:38Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:07:38Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:07:39Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.78s) FAIL test/e2e.TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 5) (4.78s) === RUN TestAPISIXSuite --- FAIL: TestAPISIXSuite (5.16s) FAIL test/e2e.TestAPISIXSuite (re-run 5) (5.16s) FAIL test/e2e === RUN TestExperimentSuite/TestExperimentWithDryRunMetrics time="2024-10-30T12:07:48Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:07:50Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:07:51Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:07:51Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:08:24Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T12:08:24Z" level=info msg="Sleeping 3s" time="2024-10-30T12:08:27Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:08:27Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:08:27Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-5khqn Pod ✔ Running 36s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ 17708f5f-2734-4616-b9ab-907df66a9e66.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:07:51Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20349628" uid: 5e873aaa-667d-47e4-94af-af6fd5413f1d spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:07:54Z" conditions: - lastTransitionTime: "2024-10-30T12:08:24Z" lastUpdateTime: "2024-10-30T12:08:24Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:08:24Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:07:51Z (x2) Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:07:51Z (x2) Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:07:51Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:07:54Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:07:54Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:07:54Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:07:54Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:08:24Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:08:24Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:08:24Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:08:27Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:08:27Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.64s) FAIL test/e2e.TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 5) (38.64s) === RUN TestExperimentSuite --- FAIL: TestExperimentSuite (39.03s) FAIL test/e2e.TestExperimentSuite (re-run 5) (39.03s) FAIL test/e2e === RUN TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:08:38Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:08:38Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 93 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc000582680, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc00050c200) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc00083bb80?, 0xc0006aa1a0?, 0xc0005bfa90?}, {0x3212d41, 0x4}, {0xc0009d7e70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc00083bb80?, 0xc0006aa1a0?, 0xc00050c200?}, {0xc0009d7e70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc000582680) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc000582680, 0xc0009850e0) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 68 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:08:38Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.54s) FAIL test/e2e.TestFunctionalSuite/TestControllerMetrics (re-run 5) (0.54s) === RUN TestFunctionalSuite time="2024-10-30T12:08:38Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" --- FAIL: TestFunctionalSuite (2.00s) FAIL test/e2e.TestFunctionalSuite (re-run 5) (2.00s) FAIL test/e2e === Skipped === SKIP: test/e2e TestAppMeshSuite (0.04s) === SKIP: test/e2e TestAWSSuite/TestALBBlueGreenUpdate (0.48s) time="2024-10-30T11:31:24Z" level=info msg="Deleting e2e-test-name=TestALBBlueGreenUpdate" time="2024-10-30T11:31:25Z" level=info msg="Deleting e2e-test-name=TestALBBlueGreenUpdate" --- SKIP: TestAWSSuite/TestALBBlueGreenUpdate (0.48s) === SKIP: test/e2e TestAWSSuite/TestALBCanaryUpdate (0.48s) time="2024-10-30T11:31:25Z" level=info msg="Deleting e2e-test-name=TestALBCanaryUpdate" time="2024-10-30T11:31:25Z" level=info msg="Deleting e2e-test-name=TestALBCanaryUpdate" --- SKIP: TestAWSSuite/TestALBCanaryUpdate (0.48s) === SKIP: test/e2e TestAWSSuite/TestALBCanaryUpdateMultiIngress (0.48s) time="2024-10-30T11:31:25Z" level=info msg="Deleting e2e-test-name=TestALBCanaryUpdateMultiIngress" time="2024-10-30T11:31:26Z" level=info msg="Deleting e2e-test-name=TestALBCanaryUpdateMultiIngress" --- SKIP: TestAWSSuite/TestALBCanaryUpdateMultiIngress (0.48s) === SKIP: test/e2e TestSMIIngressSuite (0.04s) === SKIP: test/e2e TestSMISuite (0.04s) === Failed === FAIL: test/e2e TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (10.70s) time="2024-10-30T11:30:40Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T11:30:43Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T11:30:45Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T11:30:47Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route created\n" rollout=rollout-apisix-canary time="2024-10-30T11:30:50Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T11:30:50Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T11:30:50Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (10.70s) === FAIL: test/e2e TestAPISIXSuite (44.40s) === FAIL: test/e2e TestCanarySuite/TestCanaryDynamicStableScale (22.38s) time="2024-10-30T11:37:22Z" level=info msg="Deleting e2e-test-name=TestCanaryDynamicStableScale" time="2024-10-30T11:37:24Z" level=info msg="service/dynamic-stable-scale-root created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:25Z" level=info msg="service/dynamic-stable-scale-canary created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:26Z" level=info msg="service/dynamic-stable-scale-stable created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:27Z" level=info msg="ingress.networking.k8s.io/dynamic-stable-scale-ingress created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:28Z" level=info msg="rollout.argoproj.io/dynamic-stable-scale created\n" rollout=dynamic-stable-scale time="2024-10-30T11:37:30Z" level=info msg="Marked 4 revision 1 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:30Z" level=info msg="Waiting for condition: status=Healthy" rollout=dynamic-stable-scale time="2024-10-30T11:37:31Z" level=info msg="Condition 'status=Healthy' met after 1s" rollout=dynamic-stable-scale time="2024-10-30T11:37:31Z" level=info msg="Updated rollout pod spec: 2024-10-30T11:37:31.586220824Z" rollout=dynamic-stable-scale time="2024-10-30T11:37:32Z" level=info msg="Marked 1 revision 2 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:32Z" level=info msg="Waiting for condition: status=Paused" rollout=dynamic-stable-scale time="2024-10-30T11:37:33Z" level=info msg="Condition 'status=Paused' met after 0s" rollout=dynamic-stable-scale time="2024-10-30T11:37:33Z" level=info msg="Sleeping 2s" rollout=dynamic-stable-scale time="2024-10-30T11:37:35Z" level=info msg="Pod expectation 'revision:1 pod count == 3' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:35Z" level=info msg="Pod expectation 'revision:2 pod count == 1' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:35Z" level=info msg="Promoted rollout" rollout=dynamic-stable-scale time="2024-10-30T11:37:36Z" level=info msg="Marked 2 revision 2 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:36Z" level=info msg="Waiting for condition: status.currentStepIndex=3" rollout=dynamic-stable-scale time="2024-10-30T11:37:38Z" level=info msg="Condition 'status.currentStepIndex=3' met after 1s" rollout=dynamic-stable-scale time="2024-10-30T11:37:38Z" level=info msg="Sleeping 2s" rollout=dynamic-stable-scale time="2024-10-30T11:37:40Z" level=info msg="Pod expectation 'revision:1 pod count == 1' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:40Z" level=info msg="Pod expectation 'revision:2 pod count == 3' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:40Z" level=info msg="Aborted rollout" rollout=dynamic-stable-scale time="2024-10-30T11:37:41Z" level=info msg="Marked 2 revision 1 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:41Z" level=info msg="Waiting for condition: rev=2 podcount=1" rollout=dynamic-stable-scale time="2024-10-30T11:37:43Z" level=info msg="Condition 'rev=2 podcount=1' met after 2s" rollout=dynamic-stable-scale time="2024-10-30T11:37:43Z" level=info msg="Pod expectation 'revision:1 pod count == 4' met" rollout=dynamic-stable-scale time="2024-10-30T11:37:44Z" level=info msg="Marked 1 revision 1 pods as ready" rollout=dynamic-stable-scale time="2024-10-30T11:37:44Z" level=info msg="Waiting for condition: rev=2 podcount=0" rollout=dynamic-stable-scale time="2024-10-30T11:37:44Z" level=info msg="Condition 'rev=2 podcount=0' met after 0s" rollout=dynamic-stable-scale canary_test.go:638: Expected dynamic-stable-scale-canary selector: map[app:dynamic-stable-scale rollouts-pod-template-hash:868d98995b]. Actual: map[app:dynamic-stable-scale rollouts-pod-template-hash:559649b568] Name: dynamic-stable-scale Namespace: argo-rollouts Status: ✖ Degraded Message: RolloutAborted: Rollout aborted update to revision 2 Strategy: Canary Step: 0/4 SetWeight: 0 ActualWeight: 0 Images: nginx:1.19-alpine (stable) Replicas: Desired: 4 Current: 4 Updated: 0 Ready: 4 Available: 4 NAME KIND STATUS AGE INFO ⟳ dynamic-stable-scale Rollout ✖ Degraded 16s ├──# revision:2 │ └──⧉ dynamic-stable-scale-559649b568 ReplicaSet • ScaledDown 13s canary │ └──□ dynamic-stable-scale-559649b568-pj5gg Pod ◌ Terminating 13s ready:1/1 └──# revision:1 └──⧉ dynamic-stable-scale-868d98995b ReplicaSet ✔ Healthy 16s stable ├──□ dynamic-stable-scale-868d98995b-sz5zh Pod ✔ Running 16s ready:1/1 ├──□ dynamic-stable-scale-868d98995b-5g8lv Pod ✔ Running 4s ready:1/1 ├──□ dynamic-stable-scale-868d98995b-b2wmr Pod ✔ Running 4s ready:1/1 └──□ dynamic-stable-scale-868d98995b-n4zsx Pod ✔ Running 4s ready:1/1 --- apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: annotations: rollout.argoproj.io/revision: "2" creationTimestamp: "2024-10-30T11:37:28Z" generation: 2 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestCanaryDynamicStableScale name: dynamic-stable-scale namespace: argo-rollouts resourceVersion: "20301797" uid: c0a14511-b4c0-4f42-a0bc-471fe70d235d spec: replicas: 4 selector: matchLabels: app: dynamic-stable-scale strategy: canary: canaryService: dynamic-stable-scale-canary dynamicStableScale: true stableService: dynamic-stable-scale-stable steps: - setWeight: 25 - pause: {} - setWeight: 75 - pause: {} trafficRouting: alb: ingress: dynamic-stable-scale-ingress rootService: dynamic-stable-scale-root servicePort: 80 template: metadata: annotations: update: "2024-10-30T11:37:31.586220824Z" creationTimestamp: null labels: app: dynamic-stable-scale spec: containers: - image: nginx:1.19-alpine name: dynamic-stable-scale ports: - containerPort: 80 name: http protocol: TCP resources: requests: cpu: 5m memory: 16Mi readinessGates: - conditionType: argoproj.io/e2e-readiness status: HPAReplicas: 4 abort: true abortedAt: "2024-10-30T11:37:40Z" availableReplicas: 4 blueGreen: {} canary: weights: canary: podTemplateHash: 559649b568 serviceName: dynamic-stable-scale-canary weight: 0 stable: podTemplateHash: 868d98995b serviceName: dynamic-stable-scale-stable weight: 100 conditions: - lastTransitionTime: "2024-10-30T11:37:31Z" lastUpdateTime: "2024-10-30T11:37:31Z" message: Rollout has minimum availability reason: AvailableReason status: "True" type: Available - lastTransitionTime: "2024-10-30T11:37:31Z" lastUpdateTime: "2024-10-30T11:37:31Z" message: Rollout is not healthy reason: RolloutHealthy status: "False" type: Healthy - lastTransitionTime: "2024-10-30T11:37:31Z" lastUpdateTime: "2024-10-30T11:37:31Z" message: RolloutCompleted reason: RolloutCompleted status: "False" type: Completed - lastTransitionTime: "2024-10-30T11:37:38Z" lastUpdateTime: "2024-10-30T11:37:38Z" message: Rollout is paused reason: RolloutPaused status: "True" type: Paused - lastTransitionTime: "2024-10-30T11:37:40Z" lastUpdateTime: "2024-10-30T11:37:40Z" message: Rollout aborted update to revision 2 reason: RolloutAborted status: "False" type: Progressing currentPodHash: 559649b568 currentStepHash: 7c5b668566 currentStepIndex: 0 message: 'RolloutAborted: Rollout aborted update to revision 2' observedGeneration: "2" phase: Degraded readyReplicas: 4 replicas: 4 selector: app=dynamic-stable-scale stableRS: 868d98995b 2024-10-30T11:37:28Z Normal RolloutAddedToInformer Rollout resource added to informer: argo-rollouts/dynamic-stable-scale 2024-10-30T11:37:28Z Normal RolloutNotCompleted Rollout not completed, started update to revision 1 (868d98995b) 2024-10-30T11:37:28Z Normal RolloutUpdated Rollout updated to revision 1 2024-10-30T11:37:28Z Normal NewReplicaSetCreated Created ReplicaSet dynamic-stable-scale-868d98995b (revision 1) 2024-10-30T11:37:28Z Normal PatchingALBIngress Updating Ingress `dynamic-stable-scale-ingress` to desiredWeight '0' 2024-10-30T11:37:28Z Normal TrafficWeightUpdated Traffic weight updated to 0 2024-10-30T11:37:28Z Normal ScalingReplicaSet Scaled up ReplicaSet dynamic-stable-scale-868d98995b (revision 1) from 0 to 4 2024-10-30T11:37:28Z Normal RolloutCompleted Rollout completed update to revision 1 (868d98995b): Initial deploy 2024-10-30T11:37:31Z (x2) Normal TrafficWeightUpdated Traffic weight updated 2024-10-30T11:37:31Z Normal SwitchService Switched selector for service 'dynamic-stable-scale-stable' from '' to '868d98995b' 2024-10-30T11:37:31Z Normal SwitchService Switched selector for service 'dynamic-stable-scale-canary' from '' to '868d98995b' 2024-10-30T11:37:31Z Normal RolloutNotCompleted Rollout not completed, started update to revision 2 (559649b568) 2024-10-30T11:37:31Z Normal RolloutUpdated Rollout updated to revision 2 2024-10-30T11:37:31Z Normal NewReplicaSetCreated Created ReplicaSet dynamic-stable-scale-559649b568 (revision 2) 2024-10-30T11:37:31Z Normal ScalingReplicaSet Scaled up ReplicaSet dynamic-stable-scale-559649b568 (revision 2) from 0 to 1 2024-10-30T11:37:33Z Normal SwitchService Switched selector for service 'dynamic-stable-scale-canary' from '868d98995b' to '559649b568' 2024-10-30T11:37:33Z Normal PatchingALBIngress Updating Ingress `dynamic-stable-scale-ingress` to desiredWeight '25' 2024-10-30T11:37:33Z Normal TrafficWeightUpdated Traffic weight updated from 0 to 25 2024-10-30T11:37:33Z Normal RolloutStepCompleted Rollout step 1/4 completed (setWeight: 25) 2024-10-30T11:37:33Z Normal ScalingReplicaSet Scaled down ReplicaSet dynamic-stable-scale-868d98995b (revision 1) from 4 to 3 2024-10-30T11:37:33Z Normal RolloutPaused Rollout is paused (CanaryPauseStep) 2024-10-30T11:37:35Z Normal RolloutStepCompleted Rollout step 2/4 completed (pause) 2024-10-30T11:37:35Z Normal ScalingReplicaSet Scaled up ReplicaSet dynamic-stable-scale-559649b568 (revision 2) from 1 to 3 2024-10-30T11:37:38Z Normal PatchingALBIngress Updating Ingress `dynamic-stable-scale-ingress` to desiredWeight '75' time="2024-10-30T11:37:44Z" level=info msg="Deleting e2e-test-name=TestCanaryDynamicStableScale" --- FAIL: TestCanarySuite/TestCanaryDynamicStableScale (22.38s) === FAIL: test/e2e TestCanarySuite/TestCanaryScaleDownOnAbort (12.73s) time="2024-10-30T11:40:26Z" level=info msg="Deleting e2e-test-name=TestCanaryScaleDownOnAbort" time="2024-10-30T11:40:27Z" level=info msg="service/canary-scaledowndelay-root created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:29Z" level=info msg="service/canary-scaledowndelay-canary created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:30Z" level=info msg="service/canary-scaledowndelay-stable created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:31Z" level=info msg="ingress.networking.k8s.io/canary-scaledowndelay-ingress created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:32Z" level=info msg="rollout.argoproj.io/canary-scaledownd-on-abort created\n" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:32Z" level=info msg="Waiting for condition: status=Healthy" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:35Z" level=info msg="Condition 'status=Healthy' met after 3s" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:35Z" level=info msg="Updated rollout pod spec: 2024-10-30T11:40:35.782839303Z" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:35Z" level=info msg="Waiting for condition: status=Paused" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:38Z" level=info msg="Condition 'status=Paused' met after 2s" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:38Z" level=info msg="Aborted rollout" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:38Z" level=info msg="Waiting for condition: status=Degraded" rollout=canary-scaledownd-on-abort time="2024-10-30T11:40:38Z" level=info msg="Condition 'status=Degraded' met after 0s" rollout=canary-scaledownd-on-abort canary_test.go:544: Expected canary-scaledowndelay-canary selector: map[app:canary-scaledowndelay rollouts-pod-template-hash:66597877b7]. Actual: map[app:canary-scaledowndelay rollouts-pod-template-hash:5987d8856f] Name: canary-scaledownd-on-abort Namespace: argo-rollouts Status: ✖ Degraded Message: RolloutAborted: Rollout aborted update to revision 2 Strategy: Canary Step: 0/2 SetWeight: 0 ActualWeight: 0 Images: nginx:1.19-alpine (canary, stable) Replicas: Desired: 1 Current: 2 Updated: 1 Ready: 2 Available: 2 NAME KIND STATUS AGE INFO ⟳ canary-scaledownd-on-abort Rollout ✖ Degraded 6s ├──# revision:2 │ └──⧉ canary-scaledownd-on-abort-5987d8856f ReplicaSet ✔ Healthy 3s canary,delay:0s │ └──□ canary-scaledownd-on-abort-5987d8856f-vrh9p Pod ✔ Running 3s ready:1/1 └──# revision:1 └──⧉ canary-scaledownd-on-abort-66597877b7 ReplicaSet ✔ Healthy 6s stable └──□ canary-scaledownd-on-abort-66597877b7-qz8sp Pod ✔ Running 6s ready:1/1 --- apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: annotations: rollout.argoproj.io/revision: "2" creationTimestamp: "2024-10-30T11:40:32Z" generation: 3 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestCanaryScaleDownOnAbort name: canary-scaledownd-on-abort namespace: argo-rollouts resourceVersion: "20306031" uid: bcc0b471-85ba-4c60-8b65-032d25b865b9 spec: replicas: 1 selector: matchLabels: app: canary-scaledowndelay strategy: canary: abortScaleDownDelaySeconds: 1 canaryService: canary-scaledowndelay-canary scaleDownDelayRevisionLimit: 1 stableService: canary-scaledowndelay-stable steps: - setCanaryScale: replicas: 1 - pause: {} trafficRouting: alb: ingress: canary-scaledowndelay-ingress rootService: canary-scaledowndelay-root servicePort: 80 template: metadata: annotations: update: "2024-10-30T11:40:35.782839303Z" creationTimestamp: null labels: app: canary-scaledowndelay spec: containers: - image: nginx:1.19-alpine name: canary-scaledowndelay ports: - containerPort: 80 name: http protocol: TCP resources: requests: cpu: 5m memory: 16Mi status: HPAReplicas: 2 abort: true abortedAt: "2024-10-30T11:40:38Z" availableReplicas: 2 blueGreen: {} canary: weights: canary: podTemplateHash: 5987d8856f serviceName: canary-scaledowndelay-canary weight: 0 stable: podTemplateHash: 66597877b7 serviceName: canary-scaledowndelay-stable weight: 100 conditions: - lastTransitionTime: "2024-10-30T11:40:35Z" lastUpdateTime: "2024-10-30T11:40:35Z" message: Rollout has minimum availability reason: AvailableReason status: "True" type: Available - lastTransitionTime: "2024-10-30T11:40:35Z" lastUpdateTime: "2024-10-30T11:40:35Z" message: Rollout is not healthy reason: RolloutHealthy status: "False" type: Healthy - lastTransitionTime: "2024-10-30T11:40:35Z" lastUpdateTime: "2024-10-30T11:40:35Z" message: RolloutCompleted reason: RolloutCompleted status: "False" type: Completed - lastTransitionTime: "2024-10-30T11:40:38Z" lastUpdateTime: "2024-10-30T11:40:38Z" message: Rollout is paused reason: RolloutPaused status: "True" type: Paused - lastTransitionTime: "2024-10-30T11:40:38Z" lastUpdateTime: "2024-10-30T11:40:38Z" message: Rollout aborted update to revision 2 reason: RolloutAborted status: "False" type: Progressing currentPodHash: 5987d8856f currentStepHash: 5db8f9b897 currentStepIndex: 0 message: 'RolloutAborted: Rollout aborted update to revision 2' observedGeneration: "3" phase: Degraded readyReplicas: 2 replicas: 2 selector: app=canary-scaledowndelay stableRS: 66597877b7 updatedReplicas: 1 2024-10-30T11:40:32Z Normal RolloutAddedToInformer Rollout resource added to informer: argo-rollouts/canary-scaledownd-on-abort 2024-10-30T11:40:32Z Normal RolloutNotCompleted Rollout not completed, started update to revision 1 (66597877b7) 2024-10-30T11:40:32Z Normal RolloutUpdated Rollout updated to revision 1 2024-10-30T11:40:32Z Normal NewReplicaSetCreated Created ReplicaSet canary-scaledownd-on-abort-66597877b7 (revision 1) 2024-10-30T11:40:32Z Normal PatchingALBIngress Updating Ingress `canary-scaledowndelay-ingress` to desiredWeight '0' 2024-10-30T11:40:32Z Normal TrafficWeightUpdated Traffic weight updated to 0 2024-10-30T11:40:32Z Normal ScalingReplicaSet Scaled up ReplicaSet canary-scaledownd-on-abort-66597877b7 (revision 1) from 0 to 1 2024-10-30T11:40:32Z Normal RolloutCompleted Rollout completed update to revision 1 (66597877b7): Initial deploy 2024-10-30T11:40:35Z (x2) Normal TrafficWeightUpdated Traffic weight updated 2024-10-30T11:40:35Z Normal SwitchService Switched selector for service 'canary-scaledowndelay-stable' from '' to '66597877b7' 2024-10-30T11:40:35Z Normal SwitchService Switched selector for service 'canary-scaledowndelay-canary' from '' to '66597877b7' 2024-10-30T11:40:35Z Normal RolloutNotCompleted Rollout not completed, started update to revision 2 (5987d8856f) 2024-10-30T11:40:35Z Normal RolloutUpdated Rollout updated to revision 2 2024-10-30T11:40:35Z Normal NewReplicaSetCreated Created ReplicaSet canary-scaledownd-on-abort-5987d8856f (revision 2) 2024-10-30T11:40:35Z Normal ScalingReplicaSet Scaled up ReplicaSet canary-scaledownd-on-abort-5987d8856f (revision 2) from 0 to 1 2024-10-30T11:40:38Z (x3) Normal SwitchService Switched selector for service 'canary-scaledowndelay-canary' from '66597877b7' to '5987d8856f' 2024-10-30T11:40:38Z Normal RolloutStepCompleted Rollout step 1/2 completed (setCanaryScale{replicas: 1}) 2024-10-30T11:40:38Z Normal RolloutPaused Rollout is paused (CanaryPauseStep) 2024-10-30T11:40:38Z (x3) Normal SwitchService Switched selector for service 'canary-scaledowndelay-canary' from '5987d8856f' to '66597877b7' 2024-10-30T11:40:38Z Warning RolloutAborted Rollout aborted update to revision 2 time="2024-10-30T11:40:38Z" level=info msg="Deleting e2e-test-name=TestCanaryScaleDownOnAbort" --- FAIL: TestCanarySuite/TestCanaryScaleDownOnAbort (12.73s) === FAIL: test/e2e TestCanarySuite (397.49s) time="2024-10-30T11:37:22Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" === FAIL: test/e2e TestExperimentSuite/TestExperimentWithDryRunMetrics (38.60s) time="2024-10-30T11:44:16Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T11:44:18Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T11:44:19Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T11:44:19Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T11:44:52Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T11:44:52Z" level=info msg="Sleeping 3s" time="2024-10-30T11:44:55Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T11:44:55Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T11:44:55Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-n6tsk Pod ✔ Running 36s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ fcf03914-e255-4633-bb07-15086c587af4.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T11:44:19Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20313600" uid: fa3b5e30-cc72-4d4d-aee8-8e4bf4f86d9e spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T11:44:22Z" conditions: - lastTransitionTime: "2024-10-30T11:44:52Z" lastUpdateTime: "2024-10-30T11:44:52Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T11:44:52Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T11:44:19Z (x2) Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T11:44:19Z (x2) Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T11:44:19Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T11:44:22Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T11:44:22Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T11:44:22Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T11:44:22Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T11:44:52Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T11:44:52Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T11:44:52Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T11:44:55Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.60s) === FAIL: test/e2e TestExperimentSuite (134.06s) === FAIL: test/e2e TestFunctionalSuite/TestControllerMetrics (0.49s) time="2024-10-30T11:48:51Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T11:48:51Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 6147 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc0017e9d40, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc000f69000) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc001529100?, 0xc0009327f8?, 0xc001104400?}, {0x3212d41, 0x4}, {0xc001601e70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc001529100?, 0xc0009327f8?, 0xc000f69000?}, {0xc001601e70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc0017e9d40) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc0017e9d40, 0xc000806870) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 5156 /usr/local/go/src/testing/testing.go:1648 +0x3ad --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.49s) === FAIL: test/e2e TestFunctionalSuite/TestRolloutAbortRetryPromote (124.78s) time="2024-10-30T11:49:55Z" level=info msg="Deleting e2e-test-name=TestRolloutAbortRetryPromote" time="2024-10-30T11:49:55Z" level=info msg="Event watcher started" time="2024-10-30T11:49:57Z" level=info msg="rollout.argoproj.io/abort-retry-promote created\n" rollout=abort-retry-promote time="2024-10-30T11:49:57Z" level=info msg="Waiting for condition: status=Healthy" rollout=abort-retry-promote time="2024-10-30T11:49:59Z" level=info msg="Condition 'status=Healthy' met after 2s" rollout=abort-retry-promote time="2024-10-30T11:49:59Z" level=info msg="Updated rollout pod spec: 2024-10-30T11:49:59.915577022Z" rollout=abort-retry-promote time="2024-10-30T11:49:59Z" level=info msg="Waiting for condition: status=Paused" rollout=abort-retry-promote when.go:455: timeout after 2m0s waiting for condition status=Paused time="2024-10-30T11:51:59Z" level=info msg="Event watcher stopped" rollout=abort-retry-promote Name: abort-retry-promote Namespace: argo-rollouts Status: ◌ Progressing Message: waiting for rollout spec update to be observed Strategy: Canary Step: 2/2 SetWeight: 100 ActualWeight: 100 Images: nginx:1.19-alpine (stable) Replicas: Desired: 1 Current: 1 Updated: 1 Ready: 1 Available: 1 NAME KIND STATUS AGE INFO ⟳ abort-retry-promote Rollout ◌ Progressing 2m3s └──# revision:1 └──⧉ abort-retry-promote-7fd9b5545c ReplicaSet ✔ Healthy 2m3s stable └──□ abort-retry-promote-7fd9b5545c-v79v7 Pod ✔ Running 2m3s ready:1/1 --- apiVersion: argoproj.io/v1alpha1 kind: Rollout metadata: annotations: rollout.argoproj.io/revision: "1" creationTimestamp: "2024-10-30T11:49:57Z" generation: 3 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestRolloutAbortRetryPromote name: abort-retry-promote namespace: argo-rollouts resourceVersion: "20321690" uid: 97ddbad8-8c88-4850-ada1-289d2c5dc02c spec: replicas: 1 selector: matchLabels: app: abort-retry-promote strategy: canary: steps: - setWeight: 50 - pause: duration: 3s template: metadata: annotations: update: "2024-10-30T11:49:59.915577022Z" creationTimestamp: null labels: app: abort-retry-promote spec: containers: - image: nginx:1.19-alpine name: abort-retry-promote resources: requests: cpu: 1m memory: 16Mi status: HPAReplicas: 1 availableReplicas: 1 blueGreen: {} canary: {} conditions: - lastTransitionTime: "2024-10-30T11:49:57Z" lastUpdateTime: "2024-10-30T11:49:57Z" message: RolloutCompleted reason: RolloutCompleted status: "True" type: Completed - lastTransitionTime: "2024-10-30T11:49:59Z" lastUpdateTime: "2024-10-30T11:49:59Z" message: Rollout is healthy reason: RolloutHealthy status: "True" type: Healthy - lastTransitionTime: "2024-10-30T11:49:57Z" lastUpdateTime: "2024-10-30T11:49:59Z" message: ReplicaSet "abort-retry-promote-7fd9b5545c" has successfully progressed. reason: NewReplicaSetAvailable status: "True" type: Progressing - lastTransitionTime: "2024-10-30T11:49:59Z" lastUpdateTime: "2024-10-30T11:49:59Z" message: Rollout has minimum availability reason: AvailableReason status: "True" type: Available currentPodHash: 7fd9b5545c currentStepHash: 77fbc56786 currentStepIndex: 2 observedGeneration: "2" phase: Healthy readyReplicas: 1 replicas: 1 selector: app=abort-retry-promote stableRS: 7fd9b5545c updatedReplicas: 1 2024-10-30T11:49:57Z Normal RolloutAddedToInformer Rollout resource added to informer: argo-rollouts/abort-retry-promote 2024-10-30T11:49:57Z Normal RolloutNotCompleted Rollout not completed, started update to revision 1 (7fd9b5545c) 2024-10-30T11:49:57Z Normal RolloutUpdated Rollout updated to revision 1 2024-10-30T11:49:57Z Normal NewReplicaSetCreated Created ReplicaSet abort-retry-promote-7fd9b5545c (revision 1) 2024-10-30T11:49:57Z Normal ScalingReplicaSet Scaled up ReplicaSet abort-retry-promote-7fd9b5545c (revision 1) from 0 to 1 2024-10-30T11:49:57Z Normal RolloutCompleted Rollout completed update to revision 1 (7fd9b5545c): Initial deploy time="2024-10-30T11:52:00Z" level=info msg="Deleting e2e-test-name=TestRolloutAbortRetryPromote" --- FAIL: TestFunctionalSuite/TestRolloutAbortRetryPromote (124.78s) === FAIL: test/e2e TestFunctionalSuite (560.01s) time="2024-10-30T11:46:48Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" === FAIL: test/e2e TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 1) (5.16s) time="2024-10-30T12:01:07Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:01:09Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:01:10Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:01:11Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route configured\n" rollout=rollout-apisix-canary time="2024-10-30T12:01:12Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:01:12Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:01:12Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:01:12Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (5.16s) === FAIL: test/e2e TestAPISIXSuite (re-run 1) (5.53s) === FAIL: test/e2e TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 1) (38.89s) time="2024-10-30T12:02:24Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:02:26Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:02:27Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:02:27Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:03:00Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T12:03:00Z" level=info msg="Sleeping 3s" time="2024-10-30T12:03:03Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:03:03Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:03:03Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-pv5sn Pod ✔ Running 36s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ ef43dcb9-fef6-45df-8526-f52e1699df8e.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:02:27Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20342675" uid: ed9953aa-fcab-4976-bb6c-b5dc2ab5d7cd spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:02:30Z" conditions: - lastTransitionTime: "2024-10-30T12:03:00Z" lastUpdateTime: "2024-10-30T12:03:00Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:03:00Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:02:27Z Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:02:27Z Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:02:27Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:02:30Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:02:30Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:02:30Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:02:30Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:03:00Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:03:00Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:03:00Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:03:03Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:03:03Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.89s) === FAIL: test/e2e TestExperimentSuite (re-run 1) (39.23s) === FAIL: test/e2e TestFunctionalSuite/TestControllerMetrics (re-run 1) (0.57s) time="2024-10-30T12:03:14Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:03:14Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 123 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc0001a7ba0, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc000c3c000) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc00055fa40?, 0xc0004d8270?, 0xc000759ca0?}, {0x3212d41, 0x4}, {0xc0007ffe70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc00055fa40?, 0xc0004d8270?, 0xc000c3c000?}, {0xc0007ffe70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc0001a7ba0) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc0001a7ba0, 0xc000563680) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 82 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:03:15Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.57s) === FAIL: test/e2e TestFunctionalSuite (re-run 1) (2.26s) time="2024-10-30T12:03:14Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" === FAIL: test/e2e TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 2) (5.08s) time="2024-10-30T12:03:51Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:03:52Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:03:53Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:03:54Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:03:55Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:03:55Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:03:55Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:03:56Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (5.08s) === FAIL: test/e2e TestAPISIXSuite (re-run 2) (5.42s) === FAIL: test/e2e TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 2) (38.34s) time="2024-10-30T12:04:06Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:04:07Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:04:08Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:04:08Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:04:41Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T12:04:41Z" level=info msg="Sleeping 3s" time="2024-10-30T12:04:44Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:04:44Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:04:44Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-lrkmk Pod ✔ Running 36s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ 2909fff8-b8da-42f3-b43b-c3965df56dac.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:04:08Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20344918" uid: e4adc1cb-2c41-47fc-85c1-845b82ab7368 spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:04:11Z" conditions: - lastTransitionTime: "2024-10-30T12:04:41Z" lastUpdateTime: "2024-10-30T12:04:41Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:04:41Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:04:08Z Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:04:08Z Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:04:08Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:04:11Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:04:11Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:04:11Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:04:11Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:04:41Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:04:41Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:04:41Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:04:44Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:04:44Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.34s) === FAIL: test/e2e TestExperimentSuite (re-run 2) (38.68s) === FAIL: test/e2e TestFunctionalSuite/TestControllerMetrics (re-run 2) (0.57s) time="2024-10-30T12:04:56Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:04:56Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 124 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc0001a6820, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc0001ccb00) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc000a93340?, 0xc0009ce348?, 0xc0009f7910?}, {0x3212d41, 0x4}, {0xc000845e70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc000a93340?, 0xc0009ce348?, 0xc0001ccb00?}, {0xc000845e70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc0001a6820) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc0001a6820, 0xc000847cb0) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 15 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:04:57Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.57s) === FAIL: test/e2e TestFunctionalSuite (re-run 2) (2.09s) time="2024-10-30T12:04:56Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" === FAIL: test/e2e TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 3) (4.99s) time="2024-10-30T12:05:07Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:05:08Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:05:09Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:05:10Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:05:11Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:05:11Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:05:11Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:05:12Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.99s) === FAIL: test/e2e TestAPISIXSuite (re-run 3) (5.34s) === FAIL: test/e2e TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 3) (38.59s) time="2024-10-30T12:05:21Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:05:23Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:05:24Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:05:24Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:05:57Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T12:05:57Z" level=info msg="Sleeping 3s" time="2024-10-30T12:06:00Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:06:00Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:06:00Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-x227n Pod ✔ Running 35s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ 68e50f3e-5c93-4e93-b1df-894005693883.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:05:24Z" generation: 10 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20346544" uid: fef5a489-2501-4c18-877b-a588e257b1e2 spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:05:27Z" conditions: - lastTransitionTime: "2024-10-30T12:05:57Z" lastUpdateTime: "2024-10-30T12:05:57Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:05:57Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:05:24Z Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:05:24Z Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:05:25Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:05:27Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:05:27Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:05:27Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:05:27Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:05:57Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:05:57Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:05:57Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:06:00Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:06:00Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.59s) === FAIL: test/e2e TestExperimentSuite (re-run 3) (38.95s) === FAIL: test/e2e TestFunctionalSuite/TestControllerMetrics (re-run 3) (0.54s) time="2024-10-30T12:06:11Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:06:11Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 78 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc000102b60, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc00050e200) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc00039bd80?, 0xc000a028a8?, 0xc0009bc410?}, {0x3212d41, 0x4}, {0xc000973e70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc00039bd80?, 0xc000a028a8?, 0xc00050e200?}, {0xc000973e70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc000102b60) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc000102b60, 0xc0002fba70) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 69 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:06:11Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.54s) === FAIL: test/e2e TestFunctionalSuite (re-run 3) (1.95s) time="2024-10-30T12:06:11Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" === FAIL: test/e2e TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 4) (4.79s) time="2024-10-30T12:06:21Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:06:22Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:06:23Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:06:24Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:06:25Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:06:25Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:06:25Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:06:26Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.79s) === FAIL: test/e2e TestAPISIXSuite (re-run 4) (5.13s) === FAIL: test/e2e TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 4) (37.82s) time="2024-10-30T12:06:35Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:06:36Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:06:38Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:06:38Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:07:10Z" level=info msg="Condition 'phase=Successful' met after 31s" time="2024-10-30T12:07:10Z" level=info msg="Sleeping 3s" time="2024-10-30T12:07:13Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:07:13Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:07:13Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 35s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 35s delay:26s │ └──□ experiment-with-dry-run-baseline-sp272 Pod ✔ Running 35s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ fb2207c3-76b3-4e8a-9a15-d3c8e5d5d2a4.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:06:38Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20348070" uid: 558e8113-13af-4088-a763-0b0f30a834e1 spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:06:40Z" conditions: - lastTransitionTime: "2024-10-30T12:07:10Z" lastUpdateTime: "2024-10-30T12:07:10Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:07:10Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:06:38Z (x2) Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:06:38Z (x2) Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:06:38Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:06:40Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:06:40Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:06:40Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:07:08Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:07:10Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:07:10Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:07:10Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:07:13Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:07:13Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (37.82s) === FAIL: test/e2e TestExperimentSuite (re-run 4) (38.17s) === FAIL: test/e2e TestFunctionalSuite/TestControllerMetrics (re-run 4) (0.67s) time="2024-10-30T12:07:24Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:07:24Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 73 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc000c02340, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc0005f4d00) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc0005f2840?, 0xc000816160?, 0xc0002f25a0?}, {0x3212d41, 0x4}, {0xc00085de70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc0005f2840?, 0xc000816160?, 0xc0005f4d00?}, {0xc00085de70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc000c02340) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc000c02340, 0xc000b165a0) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 14 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:07:24Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.67s) === FAIL: test/e2e TestFunctionalSuite (re-run 4) (2.20s) time="2024-10-30T12:07:24Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" === FAIL: test/e2e TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (re-run 5) (4.78s) time="2024-10-30T12:07:34Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:07:35Z" level=info msg="service/rollout-apisix-canary-canary created\n" rollout=rollout-apisix-canary time="2024-10-30T12:07:36Z" level=info msg="service/rollout-apisix-canary-stable created\n" rollout=rollout-apisix-canary time="2024-10-30T12:07:37Z" level=info msg="apisixroute.apisix.apache.org/rollouts-apisix-route unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:07:38Z" level=info msg="rollout.argoproj.io/rollout-apisix-canary unchanged\n" rollout=rollout-apisix-canary time="2024-10-30T12:07:38Z" level=info msg="Waiting for condition: status=Healthy" rollout=rollout-apisix-canary when.go:423: rollouts.argoproj.io "rollout-apisix-canary" not found time="2024-10-30T12:07:38Z" level=info msg="Deleting e2e-test-name=TestAPISIXCanarySetHeaderStep" time="2024-10-30T12:07:39Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.78s) === FAIL: test/e2e TestAPISIXSuite (re-run 5) (5.16s) === FAIL: test/e2e TestExperimentSuite/TestExperimentWithDryRunMetrics (re-run 5) (38.64s) time="2024-10-30T12:07:48Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:07:50Z" level=info msg="analysistemplate.argoproj.io/dry-run-job created\n" time="2024-10-30T12:07:51Z" level=info msg="experiment.argoproj.io/experiment-with-dry-run created\n" time="2024-10-30T12:07:51Z" level=info msg="Waiting for Experiment experiment-with-dry-run condition: phase=Successful" time="2024-10-30T12:08:24Z" level=info msg="Condition 'phase=Successful' met after 32s" time="2024-10-30T12:08:24Z" level=info msg="Sleeping 3s" time="2024-10-30T12:08:27Z" level=info msg="Expected Dry-Run Summary: Count=1, Successful=0, Failed=1, Error=0, Inconclusive=0" time="2024-10-30T12:08:27Z" level=info msg="Analysis Dry-Run Summary: Count=1, Successful=1, Failed=0, Error=0, Inconclusive=0" time="2024-10-30T12:08:27Z" level=error msg="Dry-Run Summary of AnalysisRun: 'experiment-with-dry-run-dry-run-job' doesn't match the expectations" Name: experiment-with-dry-run Namespace: argo-rollouts Status: ✔ Successful Images: nginx:1.19-alpine (Σ:baseline) NAME KIND STATUS AGE INFO Σ experiment-with-dry-run Experiment ✔ Successful 36s ├──⧉ experiment-with-dry-run-baseline ReplicaSet ✔ Healthy 36s delay:26s │ └──□ experiment-with-dry-run-baseline-5khqn Pod ✔ Running 36s ready:1/1 └──α experiment-with-dry-run-dry-run-job AnalysisRun ✔ Successful 33s ✔ 1 └──⊞ 17708f5f-2734-4616-b9ab-907df66a9e66.test-1.1 Job ✔ Successful 33s --- apiVersion: argoproj.io/v1alpha1 kind: Experiment metadata: creationTimestamp: "2024-10-30T12:07:51Z" generation: 9 labels: argo-rollouts.argoproj.io/controller-instance-id: argo-rollouts-e2e e2e-test-name: TestExperimentWithDryRunMetrics name: experiment-with-dry-run namespace: argo-rollouts resourceVersion: "20349628" uid: 5e873aaa-667d-47e4-94af-af6fd5413f1d spec: analyses: - name: dry-run-job templateName: dry-run-job dryRun: - metricName: test.* duration: 30s progressDeadlineSeconds: 30 templates: - name: baseline replicas: 1 selector: matchLabels: app: experiment-with-dry-run service: {} template: metadata: creationTimestamp: null labels: app: experiment-with-dry-run spec: containers: - image: nginx:1.19-alpine name: experiment-with-dry-run resources: requests: cpu: 1m memory: 16Mi status: analysisRuns: - analysisRun: experiment-with-dry-run-dry-run-job name: dry-run-job phase: Running availableAt: "2024-10-30T12:07:54Z" conditions: - lastTransitionTime: "2024-10-30T12:08:24Z" lastUpdateTime: "2024-10-30T12:08:24Z" message: Experiment "experiment-with-dry-run" has successfully ran and completed. reason: ExperimentCompleted status: "False" type: Progressing phase: Successful templateStatuses: - availableReplicas: 1 lastTransitionTime: "2024-10-30T12:08:24Z" name: baseline readyReplicas: 1 replicas: 1 status: Successful updatedReplicas: 1 2024-10-30T12:07:51Z (x2) Normal TemplateProgressing Template 'baseline' transitioned from -> Progressing 2024-10-30T12:07:51Z (x2) Normal ExperimentPending Experiment transitioned from -> Pending 2024-10-30T12:07:51Z Normal ScalingReplicaSet Scaled up ReplicaSet experiment-with-dry-run-baseline from 0 to 1 2024-10-30T12:07:54Z Normal TemplateRunning Template 'baseline' transitioned from Progressing -> Running 2024-10-30T12:07:54Z Normal ExperimentRunning Experiment transitioned from Pending -> Running 2024-10-30T12:07:54Z Normal AnalysisRunPending AnalysisRun 'dry-run-job' transitioned from -> Pending 2024-10-30T12:07:54Z Normal AnalysisRunRunning AnalysisRun 'dry-run-job' transitioned from -> Running 2024-10-30T12:08:24Z Normal TemplateSuccessful Template 'baseline' transitioned from Running -> Successful 2024-10-30T12:08:24Z Normal ExperimentSuccessful Experiment transitioned from Running -> Successful 2024-10-30T12:08:24Z Normal AnalysisRunTerminating Terminating dry-run-job (experiment-with-dry-run-dry-run-job) time="2024-10-30T12:08:27Z" level=info msg="Deleting e2e-test-name=TestExperimentWithDryRunMetrics" time="2024-10-30T12:08:27Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.64s) === FAIL: test/e2e TestExperimentSuite (re-run 5) (39.03s) === FAIL: test/e2e TestFunctionalSuite/TestControllerMetrics (re-run 5) (0.54s) time="2024-10-30T12:08:38Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" functional_test.go:1402: Error Trace: /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1402 Error: Received unexpected error: Get "http://localhost:8090/metrics": dial tcp [::1]:8090: connect: connection refused Test: TestFunctionalSuite/TestControllerMetrics time="2024-10-30T12:08:38Z" level=info msg="Deleting e2e-test-name=TestControllerMetrics" suite.go:87: test panicked: runtime error: invalid memory address or nil pointer dereference goroutine 93 [running]: runtime/debug.Stack() /usr/local/go/src/runtime/debug/stack.go:24 +0x5e github.com/stretchr/testify/suite.failOnPanic(0xc000582680, {0x2ea07e0, 0x4cf8420}) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:87 +0x39 github.com/stretchr/testify/suite.Run.func1.1() /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:183 +0x24c panic({0x2ea07e0?, 0x4cf8420?}) /usr/local/go/src/runtime/panic.go:914 +0x21f github.com/argoproj/argo-rollouts/test/e2e.(*FunctionalSuite).TestControllerMetrics(0xc00050c200) /tmp/tmp.58kGTG6WqB/argo-rollouts/test/e2e/functional_test.go:1403 +0xc4 reflect.Value.call({0xc00083bb80?, 0xc0006aa1a0?, 0xc0005bfa90?}, {0x3212d41, 0x4}, {0xc0009d7e70, 0x1, 0x2b4e148?}) /usr/local/go/src/reflect/value.go:596 +0xce7 reflect.Value.Call({0xc00083bb80?, 0xc0006aa1a0?, 0xc00050c200?}, {0xc0009d7e70?, 0x1207aed?, 0x4460798?}) /usr/local/go/src/reflect/value.go:380 +0xb9 github.com/stretchr/testify/suite.Run.func1(0xc000582680) /root/go/pkg/mod/github.com/stretchr/testify@v1.8.4/suite/suite.go:197 +0x467 testing.tRunner(0xc000582680, 0xc0009850e0) /usr/local/go/src/testing/testing.go:1595 +0xff created by testing.(*T).Run in goroutine 68 /usr/local/go/src/testing/testing.go:1648 +0x3ad time="2024-10-30T12:08:38Z" level=info msg="Deleting e2e-test-name" --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.54s) === FAIL: test/e2e TestFunctionalSuite (re-run 5) (2.00s) time="2024-10-30T12:08:38Z" level=info msg="analysistemplate.argoproj.io/sleep-job created\n" DONE 6 runs, 140 tests, 6 skipped, 40 failures in 3261.204s make: *** [Makefile:239: test-e2e] Error 1 + set +x ----------------------------------------------------------------- These were the tests that succeeded: PASS test/e2e PASS test/e2e PASS test/e2e PASS test/e2e.TestAPISIXSuite/TestAPISIXCanaryStep (33.42s) PASS test/e2e.TestAWSSuite (203.65s) PASS test/e2e.TestAWSSuite/TestALBExperimentStep (30.17s) PASS test/e2e.TestAWSSuite/TestALBExperimentStepMultiIngress (33.18s) PASS test/e2e.TestAWSSuite/TestALBExperimentStepNoSetWeight (24.50s) PASS test/e2e.TestAWSSuite/TestALBExperimentStepNoSetWeightMultiIngress (25.59s) PASS test/e2e.TestAWSSuite/TestALBPingPongUpdate (24.51s) PASS test/e2e.TestAWSSuite/TestALBPingPongUpdateMultiIngress (23.13s) PASS test/e2e.TestAWSSuite/TestAlbHeaderRoute (14.28s) PASS test/e2e.TestAWSSuite/TestAlbHeaderRouteMultiIngress (26.55s) PASS test/e2e.TestAnalysisSuite (379.63s) PASS test/e2e.TestAnalysisSuite/TestAnalysisWithArgs (47.26s) PASS test/e2e.TestAnalysisSuite/TestAnalysisWithSecret (34.04s) PASS test/e2e.TestAnalysisSuite/TestBackgroundAnalysisWithArgs (31.52s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenAbortAndUpdate (26.24s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenAnalysis (35.31s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenKitchenSink (52.23s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenPostPromotionFail (20.79s) PASS test/e2e.TestAnalysisSuite/TestBlueGreenPrePromotionFail (33.06s) PASS test/e2e.TestAnalysisSuite/TestCanaryBackgroundAnalysis (8.08s) PASS test/e2e.TestAnalysisSuite/TestCanaryFailInlineMultipleAnalysis (20.75s) PASS test/e2e.TestAnalysisSuite/TestCanaryInlineAnalysis (26.36s) PASS test/e2e.TestAnalysisSuite/TestCanaryInlineMultipleAnalysis (23.64s) PASS test/e2e.TestAnalysisSuite/TestMultipleAnalysis (15.51s) PASS test/e2e.TestBlueGreenSuite (153.40s) PASS test/e2e.TestBlueGreenSuite/TestBlueGreenPreviewReplicaCount (12.38s) PASS test/e2e.TestBlueGreenSuite/TestBlueGreenProgressDeadlineExceededWithPause (63.83s) PASS test/e2e.TestBlueGreenSuite/TestBlueGreenProgressDeadlineExceededWithoutPause (63.74s) PASS test/e2e.TestBlueGreenSuite/TestEphemeralMetadata (13.17s) PASS test/e2e.TestCanarySuite (re-run 1) (16.21s) PASS test/e2e.TestCanarySuite (re-run 1) (25.22s) PASS test/e2e.TestCanarySuite/TestCanaryDynamicStableScale (re-run 1) (23.71s) PASS test/e2e.TestCanarySuite/TestCanaryDynamicStableScaleRollbackToStable (18.35s) PASS test/e2e.TestCanarySuite/TestCanaryProgressDeadlineExceededWithPause (123.59s) PASS test/e2e.TestCanarySuite/TestCanaryScaleDownDelay (19.32s) PASS test/e2e.TestCanarySuite/TestCanaryScaleDownOnAbort (re-run 1) (14.58s) PASS test/e2e.TestCanarySuite/TestCanaryScaleDownOnAbortNoTrafficRouting (14.88s) PASS test/e2e.TestCanarySuite/TestCanarySetCanaryScale (17.75s) PASS test/e2e.TestCanarySuite/TestCanaryUnScaleDownOnAbort (13.83s) PASS test/e2e.TestCanarySuite/TestCanaryWithPausedRollout (101.58s) PASS test/e2e.TestCanarySuite/TestEphemeralMetadata (12.58s) PASS test/e2e.TestCanarySuite/TestReduceWeightAndHonorMaxUnavailable (35.13s) PASS test/e2e.TestCanarySuite/TestRolloutScalingDuringUpdate (5.34s) PASS test/e2e.TestCanarySuite/TestRolloutScalingWhenPaused (10.75s) PASS test/e2e.TestCanarySuite/TestRolloutWithMaxSurgeScalingDuringUpdate (5.42s) PASS test/e2e.TestExperimentSuite/TestExperimentWithMeasurementRetentionMetrics (37.85s) PASS test/e2e.TestExperimentSuite/TestExperimentWithMultiportServiceAndScaleDownDelay (19.06s) PASS test/e2e.TestExperimentSuite/TestExperimentWithServiceAndScaleDownDelay (19.07s) PASS test/e2e.TestExperimentSuite/TestExperimentWithServiceNameAndScaleDownDelay (18.97s) PASS test/e2e.TestExperimentSuite/TestRolloutWithExperimentAndAnalysis (16.56s) PASS test/e2e.TestFunctionalSuite (re-run 1) (15.45s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenExceedProgressDeadlineAbort (12.04s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenPreviewReplicaCountPromoteFull (12.85s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenPromoteFull (8.94s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenScaleDownDelay (17.40s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenScaleDownOnAbort (12.95s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenToCanary (9.04s) PASS test/e2e.TestFunctionalSuite/TestBlueGreenUpdate (9.37s) PASS test/e2e.TestFunctionalSuite/TestCanaryPromoteFull (11.96s) PASS test/e2e.TestFunctionalSuite/TestCompleteRolloutRestart (23.68s) PASS test/e2e.TestFunctionalSuite/TestContainerResourceFormats (4.83s) PASS test/e2e.TestFunctionalSuite/TestFixInvalidSpec (21.62s) PASS test/e2e.TestFunctionalSuite/TestKubectlWaitForCompleted (23.50s) PASS test/e2e.TestFunctionalSuite/TestKubectlWaitForPaused (8.61s) PASS test/e2e.TestFunctionalSuite/TestMalformedRollout (5.10s) PASS test/e2e.TestFunctionalSuite/TestMalformedRolloutEphemeralCtr (4.85s) PASS test/e2e.TestFunctionalSuite/TestRolloutAbortRetryPromote (re-run 1) (13.95s) PASS test/e2e.TestFunctionalSuite/TestRolloutBadToBadUpdate (22.94s) PASS test/e2e.TestFunctionalSuite/TestRolloutBadToGoodSetWeightUpdate (26.92s) PASS test/e2e.TestFunctionalSuite/TestRolloutBadToGoodUpdate (24.18s) PASS test/e2e.TestFunctionalSuite/TestRolloutGoodToBadUpdate (26.93s) PASS test/e2e.TestFunctionalSuite/TestRolloutPDBRestart (49.84s) PASS test/e2e.TestFunctionalSuite/TestRolloutPauseDurationGreaterThanProgressDeadlineSeconds (41.98s) PASS test/e2e.TestFunctionalSuite/TestRolloutPodVolumesItemsSpec (10.60s) PASS test/e2e.TestFunctionalSuite/TestRolloutRestart (21.15s) PASS test/e2e.TestFunctionalSuite/TestWorkloadRef (19.26s) PASS test/e2e.TestFunctionalSuite/TestWorkloadRefTemplate (2.78s) PASS test/e2e.TestHeaderRoutingSuite (20.12s) PASS test/e2e.TestHeaderRoutingSuite/TestIstioHostHeaderRoute (19.83s) PASS test/e2e.TestIstioSuite (143.30s) PASS test/e2e.TestIstioSuite/TestIstioAbortUpdate (10.96s) PASS test/e2e.TestIstioSuite/TestIstioAbortUpdateDeleteAllCanaryPods (14.88s) PASS test/e2e.TestIstioSuite/TestIstioHostSplit (23.54s) PASS test/e2e.TestIstioSuite/TestIstioHostSplitExperimentStep (21.02s) PASS test/e2e.TestIstioSuite/TestIstioHostSplitOnlyTls (12.10s) PASS test/e2e.TestIstioSuite/TestIstioSubsetSplit (13.80s) PASS test/e2e.TestIstioSuite/TestIstioSubsetSplitExperimentStep (22.23s) PASS test/e2e.TestIstioSuite/TestIstioSubsetSplitSingleRoute (11.58s) PASS test/e2e.TestIstioSuite/TestIstioUpdateInMiddleZeroCanaryReplicas (12.91s) PASS test/e2e.TestMirrorRouteSuite (21.57s) PASS test/e2e.TestMirrorRouteSuite/TestIstioHostMirrorRoute (21.29s) PASS test/e2e.TestRollbackSuite (104.77s) PASS test/e2e.TestRollbackSuite/TestRollbackAnalysisOutsideWindow (57.30s) PASS test/e2e.TestRollbackSuite/TestRollbackAnalysisWithinWindow (44.77s) These were the tests that failed: --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (10.70s) --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.78s) --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.79s) --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (4.99s) --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (5.08s) --- FAIL: TestAPISIXSuite/TestAPISIXCanarySetHeaderStep (5.16s) --- FAIL: TestCanarySuite/TestCanaryDynamicStableScale (22.38s) --- FAIL: TestCanarySuite/TestCanaryScaleDownOnAbort (12.73s) --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (37.82s) --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.34s) --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.59s) --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.60s) --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.64s) --- FAIL: TestExperimentSuite/TestExperimentWithDryRunMetrics (38.89s) --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.49s) --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.54s) --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.57s) --- FAIL: TestFunctionalSuite/TestControllerMetrics (0.67s) --- FAIL: TestFunctionalSuite/TestRolloutAbortRetryPromote (124.78s) E2E tests file is complete: DONE 6 runs, 140 tests, 6 skipped, 40 failures in 3261.204s * SUCCESS: No unexpected errors occurred. * Cleaning up hack/run-upstream-argo-rollouts-e2e-tests.sh: line 7: killall: command not found hack/run-upstream-argo-rollouts-e2e-tests.sh: line 8: killall: command not found + [[ 0 != 0 ]] + kubectl delete ns argo-rollouts namespace "argo-rollouts" deleted + kubectl wait --timeout=5m --for=delete namespace/argo-rollouts + kubectl create ns argo-rollouts namespace/argo-rollouts created + kubectl config set-context --current --namespace=argo-rollouts Context "default/api-nas-412-ocp-gitops-qe-com:6443/kube:admin" modified. + cat + kubectl apply -f /tmp/tmp.hNjWJyuQmo/rollout-manager.yaml rolloutmanager.argoproj.io/argo-rollout created + cd /tmp/tmp.hNjWJyuQmo + git clone https://github.com/argoproj-labs/rollouts-plugin-trafficrouter-openshift Cloning into 'rollouts-plugin-trafficrouter-openshift'... + cd /tmp/tmp.hNjWJyuQmo/rollouts-plugin-trafficrouter-openshift + git checkout 8b4125a7f9ecffb0247df91a4c890f88c0c523b7 Note: switching to '8b4125a7f9ecffb0247df91a4c890f88c0c523b7'. You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch. If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example: git switch -c Or undo this operation with: git switch - Turn off this advice by setting config variable advice.detachedHead to false HEAD is now at 8b4125a Switch lint to use Makefile, fix go.mod + make test-e2e go test -v -p=1 -timeout=20m -race -count=1 -coverprofile=coverage.out ./tests/e2e go: downloading go1.22.2 (linux/amd64) go: downloading k8s.io/client-go v0.26.3 go: downloading github.com/onsi/ginkgo/v2 v2.19.0 go: downloading github.com/openshift/api v0.0.0-20230417092139-1b2161d23365 go: downloading k8s.io/apimachinery v0.26.3 go: downloading github.com/argoproj/argo-rollouts v1.6.6 go: downloading github.com/onsi/gomega v1.33.1 go: downloading github.com/openshift/client-go v0.0.0-20230419131419-497c7032c581 go: downloading k8s.io/api v0.26.3 go: downloading github.com/google/go-cmp v0.6.0 go: downloading github.com/go-logr/logr v1.4.1 go: downloading k8s.io/utils v0.0.0-20230726121419-3b25d923346b go: downloading golang.org/x/net v0.25.0 go: downloading golang.org/x/sys v0.20.0 go: downloading k8s.io/klog/v2 v2.110.1 go: downloading sigs.k8s.io/structured-merge-diff/v4 v4.4.1 go: downloading golang.org/x/text v0.15.0 go: downloading k8s.io/cli-runtime v0.26.3 go: downloading k8s.io/kubernetes v1.26.3 go: downloading google.golang.org/grpc v1.57.1 go: downloading golang.org/x/term v0.20.0 go: downloading sigs.k8s.io/yaml v1.4.0 go: downloading github.com/xlab/treeprint v1.2.0 go: downloading k8s.io/apiserver v0.26.3 go: downloading k8s.io/component-base v0.26.3 go: downloading k8s.io/component-helpers v0.26.3 === RUN TestPlugin Running Suite: Plugin e2e Tests Suite - /tmp/tmp.hNjWJyuQmo/rollouts-plugin-trafficrouter-openshift/tests/e2e ============================================================================================================= Random Seed: 1730290476 Will run 3 of 3 specs service/rollouts-demo-stable created service/rollouts-demo-canary created route.route.openshift.io/rollouts-demo created rollout.argoproj.io/rollouts-demo created Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy W1030 12:14:50.677989 77418 warnings.go:70] unknown field "spec.template.metadata.creationTimestamp" Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy •service/rollouts-demo-stable created service/rollouts-demo-canary created route.route.openshift.io/rollouts-demo-prod created route.route.openshift.io/rollouts-demo-stage created rollout.argoproj.io/rollouts-demo created Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy W1030 12:15:28.325357 77418 warnings.go:70] unknown field "spec.template.metadata.creationTimestamp" Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Canary weight mismatch: got 100, want 0 •service/rollouts-demo-stable created service/rollouts-demo-canary created route.route.openshift.io/rollouts-demo created analysistemplate.argoproj.io/sample-template created rollout.argoproj.io/rollouts-demo created Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy W1030 12:16:01.737062 77418 warnings.go:70] unknown field "spec.template.metadata.creationTimestamp" Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Paused Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy Rollout phase mismatch, got Progressing, want Healthy • Ran 3 of 3 Specs in 115.941 seconds SUCCESS! -- 3 Passed | 0 Failed | 0 Pending | 0 Skipped --- PASS: TestPlugin (115.94s) PASS coverage: [no statements] ok github.com/argoproj-labs/rollouts-plugin-trafficrouter-openshift/tests/e2e 117.060s coverage: [no statements] + [[ 0 != 0 ]] + oc patch subscription openshift-gitops-operator -n openshift-operators --type json --patch '[{"op": "remove", "path": "/spec/config"}]' subscription.operators.coreos.com/openshift-gitops-operator patched + update_result + [[ 2 != 0 ]] + echo test-argo-rollouts + tee /install-dir/nas-412-result.txt test-argo-rollouts + exit 0 undefined step-run-kuttl-e2e-tests $RUN_KUTTL_TESTS is set to false, skipping the execution 0 http://uploader.devtools-gitops--pipeline.svc.cluster.local:8080/CI/z-stream-test-pipeline/241030-110234/ undefined step-run-argocd-e2e-tests $RUN_ARGOCD_E2E_TESTS is set to false, skipping the execution 0 http://uploader.devtools-gitops--pipeline.svc.cluster.local:8080/CI/z-stream-test-pipeline/241030-110234/ undefined step-evaluate-failed-tasks Argo rollouts test execution has failed with exit code 2 Refer to test-argo-rollouts task undefined step-send-slack-notification ok