Uploaded image for project: 'Red Hat 3scale API Management'
  1. Red Hat 3scale API Management
  2. THREESCALE-3535

[NIGHTLIES-405] Zync is stuck in infinite loop (doesn't create routes)

XMLWordPrintable

    • Icon: Bug Bug
    • Resolution: Cannot Reproduce
    • Icon: Blocker Blocker
    • None
    • None
    • System, Zync
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Not Started
    • Hide

      There is infinite loop in zync so it generates same error whole time. This is some of it:

         (4.2ms)  BEGIN
         (0.7ms)  SAVEPOINT active_record_1
        Tenant Load (6.1ms)  SELECT  "tenants".* FROM "tenants" WHERE "tenants"."id" = $1 LIMIT $2  [["id", 2], ["LIMIT", 1]]
        Proxy Load (1.8ms)  SELECT  "proxies".* FROM "proxies" WHERE "proxies"."id" = $1 AND "proxies"."tenant_id" = $2 AND "proxies"."service_id" = $3 AND "proxies"."tenant_id" = $4 LIMIT $5  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["tenant_id", 2], ["LIMIT", 1]]
        Tenant Load (8.9ms)  SELECT  "tenants".* FROM "tenants" WHERE "tenants"."id" = $1 LIMIT $2  [["id", 2], ["LIMIT", 1]]
         (2.0ms)  COMMIT
        Tenant Load (5.6ms)  SELECT  "tenants".* FROM "tenants" WHERE "tenants"."id" = $1 LIMIT $2  [["id", 2], ["LIMIT", 1]]
        Service Load (1.1ms)  SELECT  "services".* FROM "services" WHERE "services"."id" = $1 LIMIT $2  [["id", 190], ["LIMIT", 1]]
      {"method":"PUT","path":"/tenant","format":"json","controller":"TenantsController","action":"update","status":201,"duration":17.21,"view":0.25,"db":9.78,"params":{"id":2,"endpoint":"https://3scale-admin.n5.apps.ocp.api-qe.eng.rdu2.redhat.com/","access_token":"[FILTERED]","tenant":{"id":2,"endpoint":"https://3scale-admin.n5.apps.ocp.api-qe.eng.rdu2.redhat.com/","access_token":"[FILTERED ]"}}}
         (1.2ms)  BEGIN
         (1.3ms)  BEGIN
        Proxy Create (1.5ms)  INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["created_at", "2019-09-23 15:25:59.552469"], ["updated_at", "2019-09-23 15:25:59.552469"]]
         (0.6ms)  SAVEPOINT active_record_1
         (0.2ms)  SAVEPOINT active_record_1
         (1.3ms)  ROLLBACK TO SAVEPOINT active_record_1
      [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "proxies_pkey"
      DETAIL:  Key (id)=(190) already exists.
      : INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
        Proxy Load (0.5ms)  SELECT  "proxies".* FROM "proxies" WHERE "proxies"."id" = $1 AND "proxies"."tenant_id" = $2 AND "proxies"."service_id" = $3 AND "proxies"."tenant_id" = $4 LIMIT $5  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["tenant_id", 2], ["LIMIT", 1]]
        Proxy Load (0.3ms)  SELECT  "proxies".* FROM "proxies" WHERE "proxies"."id" = $1 AND "proxies"."tenant_id" = $2 AND "proxies"."service_id" = $3 AND "proxies"."tenant_id" = $4 LIMIT $5  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["tenant_id", 2], ["LIMIT", 1]]
         (0.4ms)  SAVEPOINT active_record_1
        Service Load (1.9ms)  SELECT  "services".* FROM "services" WHERE "services"."id" = $1 LIMIT $2  [["id", 190], ["LIMIT", 1]]
        Service Load (1.3ms)  SELECT  "services".* FROM "services" WHERE "services"."id" = $1 LIMIT $2  [["id", 190], ["LIMIT", 1]]
        Proxy Load (0.6ms)  SELECT  "proxies".* FROM "proxies" WHERE "proxies"."id" = $1 AND "proxies"."tenant_id" = $2 AND "proxies"."service_id" = $3 AND "proxies"."tenant_id" = $4 LIMIT $5  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["tenant_id", 2], ["LIMIT", 1]]
        Proxy Create (2.6ms)  INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["created_at", "2019-09-23 15:25:59.563394"], ["updated_at", "2019-09-23 15:25:59.563394"]]
        Service Load (0.6ms)  SELECT  "services".* FROM "services" WHERE "services"."id" = $1 LIMIT $2  [["id", 190], ["LIMIT", 1]]
        Proxy Create (2.2ms)  INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["created_at", "2019-09-23 15:25:59.564368"], ["updated_at", "2019-09-23 15:25:59.564368"]]
         (1.4ms)  ROLLBACK TO SAVEPOINT active_record_1
         (0.2ms)  ROLLBACK TO SAVEPOINT active_record_1
      [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "proxies_pkey"
      DETAIL:  Key (id)=(190) already exists.
      : INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
      [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "proxies_pkey"
      DETAIL:  Key (id)=(190) already exists.
      : INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
        Proxy Create (1.4ms)  INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["created_at", "2019-09-23 15:25:59.567756"], ["updated_at", "2019-09-23 15:25:59.567756"]]
         (0.8ms)  SAVEPOINT active_record_1
         (0.2ms)  SAVEPOINT active_record_1
         (1.0ms)  ROLLBACK TO SAVEPOINT active_record_1
        Proxy Load (1.0ms)  SELECT  "proxies".* FROM "proxies" WHERE "proxies"."id" = $1 AND "proxies"."tenant_id" = $2 AND "proxies"."service_id" = $3 AND "proxies"."tenant_id" = $4 LIMIT $5  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["tenant_id", 2], ["LIMIT", 1]]
      [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "proxies_pkey"
      DETAIL:  Key (id)=(190) already exists.
      : INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
        Proxy Load (1.5ms)  SELECT  "proxies".* FROM "proxies" WHERE "proxies"."id" = $1 AND "proxies"."tenant_id" = $2 AND "proxies"."service_id" = $3 AND "proxies"."tenant_id" = $4 LIMIT $5  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["tenant_id", 2], ["LIMIT", 1]]
        Service Load (2.8ms)  SELECT  "services".* FROM "services" WHERE "services"."id" = $1 LIMIT $2  [["id", 190], ["LIMIT", 1]]
        Service Load (1.2ms)  SELECT  "services".* FROM "services" WHERE "services"."id" = $1 LIMIT $2  [["id", 190], ["LIMIT", 1]]
         (3.4ms)  ROLLBACK
      {"method":"PUT","path":"/notification","format":"json","controller":"NotificationsController","action":"update","status":422,"duration":46.9,"view":0.16,"db":18.27,"params":{"type":"Proxy","id":190,"parent_event_id":"858e0b2a-8582-4d0b-aea1-f3c2aea51e9d","parent_event_type":"ZyncEvent","tenant_id":2,"service_id":190,"notification":{"id":190,"tenant_id":2}}}
        Proxy Create (2.6ms)  INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["created_at", "2019-09-23 15:25:59.576840"], ["updated_at", "2019-09-23 15:25:59.576840"]]
         (2.0ms)  ROLLBACK TO SAVEPOINT active_record_1
        Proxy Create (4.2ms)  INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"  [["id", 190], ["tenant_id", 2], ["service_id", 190], ["created_at", "2019-09-23 15:25:59.577789"], ["updated_at", "2019-09-23 15:25:59.577789"]]
      [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "proxies_pkey"
      DETAIL:  Key (id)=(190) already exists.
      : INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
         (1.4ms)  ROLLBACK TO SAVEPOINT active_record_1
      [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR:  duplicate key value violates unique constraint "proxies_pkey"
      DETAIL:  Key (id)=(190) already exists.
      : INSERT INTO "proxies" ("id", "tenant_id", "service_id", "created_at", "updated_at") VALUES ($1, $2, $3, $4, $5) RETURNING "id"
        Tenant Load (3.7ms)  SELECT  "tenants".* FROM "tenants" WHERE "tenants"."id" = $1 LIMIT $2  [["id", 2], ["LIMIT", 1]]
         (2.4ms)  ROLLBACK
         (4.3ms)  ROLLBACK
         (1.9ms)  BEGIN
      

      Proxy of the service with ID 190 in the attachments.

      When I deleted service with ID 190, it wouldn't stop the loop. Redeploying zync deployment config didn't solve the problem either. After the redeploy I tried to create a new product which started the loop with the same logs (service id 190).

      Scaling pods of zync, zync-que and zync-database to 0 and back to 1 didn't work.

      Show
      There is infinite loop in zync so it generates same error whole time. This is some of it: (4.2ms) BEGIN (0.7ms) SAVEPOINT active_record_1 Tenant Load (6.1ms) SELECT "tenants" .* FROM "tenants" WHERE "tenants" . "id" = $1 LIMIT $2 [[ "id" , 2], [ "LIMIT" , 1]] Proxy Load (1.8ms) SELECT "proxies" .* FROM "proxies" WHERE "proxies" . "id" = $1 AND "proxies" . "tenant_id" = $2 AND "proxies" . "service_id" = $3 AND "proxies" . "tenant_id" = $4 LIMIT $5 [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "tenant_id" , 2], [ "LIMIT" , 1]] Tenant Load (8.9ms) SELECT "tenants" .* FROM "tenants" WHERE "tenants" . "id" = $1 LIMIT $2 [[ "id" , 2], [ "LIMIT" , 1]] (2.0ms) COMMIT Tenant Load (5.6ms) SELECT "tenants" .* FROM "tenants" WHERE "tenants" . "id" = $1 LIMIT $2 [[ "id" , 2], [ "LIMIT" , 1]] Service Load (1.1ms) SELECT "services" .* FROM "services" WHERE "services" . "id" = $1 LIMIT $2 [[ "id" , 190], [ "LIMIT" , 1]] { "method" : "PUT" , "path" : "/tenant" , "format" : "json" , "controller" : "TenantsController" , "action" : "update" , "status" :201, "duration" :17.21, "view" :0.25, "db" :9.78, "params" :{ "id" :2, "endpoint" : "https: //3scale-admin.n5.apps.ocp.api-qe.eng.rdu2.redhat.com/" , "access_token" : "[FILTERED]" , "tenant" :{ "id" :2, "endpoint" : "https://3scale-admin.n5.apps.ocp.api-qe.eng.rdu2.redhat.com/" , "access_token" : "[FILTERED ]" }}} (1.2ms) BEGIN (1.3ms) BEGIN Proxy Create (1.5ms) INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "created_at" , "2019-09-23 15:25:59.552469" ], [ "updated_at" , "2019-09-23 15:25:59.552469" ]] (0.6ms) SAVEPOINT active_record_1 (0.2ms) SAVEPOINT active_record_1 (1.3ms) ROLLBACK TO SAVEPOINT active_record_1 [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "proxies_pkey" DETAIL: Key (id)=(190) already exists. : INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" Proxy Load (0.5ms) SELECT "proxies" .* FROM "proxies" WHERE "proxies" . "id" = $1 AND "proxies" . "tenant_id" = $2 AND "proxies" . "service_id" = $3 AND "proxies" . "tenant_id" = $4 LIMIT $5 [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "tenant_id" , 2], [ "LIMIT" , 1]] Proxy Load (0.3ms) SELECT "proxies" .* FROM "proxies" WHERE "proxies" . "id" = $1 AND "proxies" . "tenant_id" = $2 AND "proxies" . "service_id" = $3 AND "proxies" . "tenant_id" = $4 LIMIT $5 [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "tenant_id" , 2], [ "LIMIT" , 1]] (0.4ms) SAVEPOINT active_record_1 Service Load (1.9ms) SELECT "services" .* FROM "services" WHERE "services" . "id" = $1 LIMIT $2 [[ "id" , 190], [ "LIMIT" , 1]] Service Load (1.3ms) SELECT "services" .* FROM "services" WHERE "services" . "id" = $1 LIMIT $2 [[ "id" , 190], [ "LIMIT" , 1]] Proxy Load (0.6ms) SELECT "proxies" .* FROM "proxies" WHERE "proxies" . "id" = $1 AND "proxies" . "tenant_id" = $2 AND "proxies" . "service_id" = $3 AND "proxies" . "tenant_id" = $4 LIMIT $5 [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "tenant_id" , 2], [ "LIMIT" , 1]] Proxy Create (2.6ms) INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "created_at" , "2019-09-23 15:25:59.563394" ], [ "updated_at" , "2019-09-23 15:25:59.563394" ]] Service Load (0.6ms) SELECT "services" .* FROM "services" WHERE "services" . "id" = $1 LIMIT $2 [[ "id" , 190], [ "LIMIT" , 1]] Proxy Create (2.2ms) INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "created_at" , "2019-09-23 15:25:59.564368" ], [ "updated_at" , "2019-09-23 15:25:59.564368" ]] (1.4ms) ROLLBACK TO SAVEPOINT active_record_1 (0.2ms) ROLLBACK TO SAVEPOINT active_record_1 [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "proxies_pkey" DETAIL: Key (id)=(190) already exists. : INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "proxies_pkey" DETAIL: Key (id)=(190) already exists. : INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" Proxy Create (1.4ms) INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "created_at" , "2019-09-23 15:25:59.567756" ], [ "updated_at" , "2019-09-23 15:25:59.567756" ]] (0.8ms) SAVEPOINT active_record_1 (0.2ms) SAVEPOINT active_record_1 (1.0ms) ROLLBACK TO SAVEPOINT active_record_1 Proxy Load (1.0ms) SELECT "proxies" .* FROM "proxies" WHERE "proxies" . "id" = $1 AND "proxies" . "tenant_id" = $2 AND "proxies" . "service_id" = $3 AND "proxies" . "tenant_id" = $4 LIMIT $5 [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "tenant_id" , 2], [ "LIMIT" , 1]] [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "proxies_pkey" DETAIL: Key (id)=(190) already exists. : INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" Proxy Load (1.5ms) SELECT "proxies" .* FROM "proxies" WHERE "proxies" . "id" = $1 AND "proxies" . "tenant_id" = $2 AND "proxies" . "service_id" = $3 AND "proxies" . "tenant_id" = $4 LIMIT $5 [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "tenant_id" , 2], [ "LIMIT" , 1]] Service Load (2.8ms) SELECT "services" .* FROM "services" WHERE "services" . "id" = $1 LIMIT $2 [[ "id" , 190], [ "LIMIT" , 1]] Service Load (1.2ms) SELECT "services" .* FROM "services" WHERE "services" . "id" = $1 LIMIT $2 [[ "id" , 190], [ "LIMIT" , 1]] (3.4ms) ROLLBACK { "method" : "PUT" , "path" : "/notification" , "format" : "json" , "controller" : "NotificationsController" , "action" : "update" , "status" :422, "duration" :46.9, "view" :0.16, "db" :18.27, "params" :{ "type" : "Proxy" , "id" :190, "parent_event_id" : "858e0b2a-8582-4d0b-aea1-f3c2aea51e9d" , "parent_event_type" : "ZyncEvent" , "tenant_id" :2, "service_id" :190, "notification" :{ "id" :190, "tenant_id" :2}}} Proxy Create (2.6ms) INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "created_at" , "2019-09-23 15:25:59.576840" ], [ "updated_at" , "2019-09-23 15:25:59.576840" ]] (2.0ms) ROLLBACK TO SAVEPOINT active_record_1 Proxy Create (4.2ms) INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" [[ "id" , 190], [ "tenant_id" , 2], [ "service_id" , 190], [ "created_at" , "2019-09-23 15:25:59.577789" ], [ "updated_at" , "2019-09-23 15:25:59.577789" ]] [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "proxies_pkey" DETAIL: Key (id)=(190) already exists. : INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" (1.4ms) ROLLBACK TO SAVEPOINT active_record_1 [Model][#``{error.class}] Retrying after PG::UniqueViolation: ERROR: duplicate key value violates unique constraint "proxies_pkey" DETAIL: Key (id)=(190) already exists. : INSERT INTO "proxies" ( "id" , "tenant_id" , "service_id" , "created_at" , "updated_at" ) VALUES ($1, $2, $3, $4, $5) RETURNING "id" Tenant Load (3.7ms) SELECT "tenants" .* FROM "tenants" WHERE "tenants" . "id" = $1 LIMIT $2 [[ "id" , 2], [ "LIMIT" , 1]] (2.4ms) ROLLBACK (4.3ms) ROLLBACK (1.9ms) BEGIN Proxy of the service with ID 190 in the attachments. When I deleted service with ID 190, it wouldn't stop the loop. Redeploying zync deployment config didn't solve the problem either. After the redeploy I tried to create a new product which started the loop with the same logs (service id 190). Scaling pods of zync, zync-que and zync-database to 0 and back to 1 didn't work.
    • 3scale 2019-09-30, 3scale 2019-10-14

      I was running some of the QE tests against nightlies-405 with APIAP. Testsuite was not changed to the integration with APIAP. It is still using 3scale endpoints for creating a service (used in 2.6).

            Unassigned Unassigned
            jsmadis Jakub Smadis (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Created:
              Updated:
              Resolved: