Uploaded image for project: 'Hybrid Cloud Console'
  1. Hybrid Cloud Console
  2. RHCLOUD-42867

Web-rca crashing: panic when updating incident via slack

XMLWordPrintable

    • False
    • Hide

      None

      Show
      None
    • False
    • None
    • Unset
    • None

      App crashes with nil pointer dereference in updateIncidentWorkingTime()when users update incident status through Slack modals. The crash occurs in the working time calculation logic, likely due to missing nil checks on timestamp fields (ResolvedAt, LastOpenedAt). This causes pod restarts and service disruption.

      Guess for how to reproduce based on the error message:

      1. Open an incident in Slack with following conditions:
      -> Status is currently or NewOngoing
       -> field is LastOpenedAtnil (this is the bug) so don't click on the link at all to go to the ui
      2. Use the Slack status update modal to change the incident status to Resolved
      3. The crash occurs when the code tries to calculate working time

      Pod log:

       

      I1017 01:44:29.224313 1 logger.go:101] [opid=34AnuvYcH2DSFkVE2HY87IxPwni] Handling event type=event_callback/message in channel CCX9DB894: Cannot get user for SlackId=U01EYSUFX2T
       
      I1017 01:44:47.003595 1 logger.go:101] SlackChannelCloser archiving ITN-2025-00202 channel: C09B2JUA0AK
       
      I1017 01:44:47.028492 1 logger.go:101] SlackChannelCloser archiving error: OCM-RCA-31: Cannot perform Slack actions. Slack bot not associated with incident channel
       
      I1017 01:44:52.384137 1 logger.go:101] [opid=34AnxjbUdWDmwB7r75Q3Pl7Wlnv] {"request_method":"POST","request_url":"/api/web-rca/v1/slack/interactive-endpoint","request_remote_ip":"10.131.0.40:60896"}
       
      I1017 01:44:52.385978 1 logger.go:101] [opid=34AnxjbUdWDmwB7r75Q3Pl7Wlnv] {"response_status":200,"elapsed":"1.80847ms"}
       
      panic: runtime error: invalid memory address or nil pointer dereference
       
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2143a9c]
       
       
       
      goroutine 3701 [running]:
       
      gitlab.cee.redhat.com/service/web-rca/pkg/services/domains.updateIncidentWorkingTime({
      {0xc000aa062b?, 0x2f24768?}
      
      , {0xc000aa0615?, 0xa0?}}, 0xc00071dc08)
       
      gitlab.cee.redhat.com/service/web-rca/pkg/services/domains/incident.go:1774 +0x9c
       
      gitlab.cee.redhat.com/service/web-rca/pkg/services/domains.(*ServiceDomain).UpdateIncident(0xc0006ce008, {0x2f24768, 0xc000b0f830}, 0xc00071dc08, 0xc001366b40)
       
      gitlab.cee.redhat.com/service/web-rca/pkg/services/domains/incident.go:1426 +0x565
       
      gitlab.cee.redhat.com/service/web-rca/pkg/slack/modals.(*slackModalIncidentStatusUpdate).Submit(0xc000462b60, {0x2f24768, 0xc000b0f830})
       
      gitlab.cee.redhat.com/service/web-rca/pkg/slack/modals/incident_status_update_modal.go:141 +0x46f
       
      gitlab.cee.redhat.com/service/web-rca/pkg/slack/modals.HandleCallback({0x2f24768, 0xc000b0f830}, 0xc000e9be00)
       
      gitlab.cee.redhat.com/service/web-rca/pkg/slack/modals/slack_modal.go:522 +0x36a
       
      gitlab.cee.redhat.com/service/web-rca/pkg/handlers.slackHandler.Interactive.func1({
      {0xc00116d170, 0xf}, {0xc00069e7f8, 0x18}, {0x0, 0x0}, {0x0, 0x0}, {0xc0006c7040, 0x39}, ...})
       
      gitlab.cee.redhat.com/service/web-rca/pkg/handlers/slack.go:272 +0x91c
       
      created by gitlab.cee.redhat.com/service/web-rca/pkg/handlers.slackHandler.Interactive in goroutine 3692
      I1017 01:41:46.889111 1 logger.go:101] Slack AuthCleaner started
      I1017 01:41:46.889445 1 logger.go:101] SlackAuthCleaner started. Token validity 1h0m0s, check interval 10m0s
      I1017 01:41:46.889652 1 logger.go:101] WebSocketAuthCleaner started with stateHold 10s
      I1017 01:41:46.905784 1 logger.go:101] WSHub: START
      I1017 01:42:04.569734 1 logger.go:101] [opid=34AnchOasnQI5xGVwS2YMMTheWJ] {"request_method":"GET","request_url":"/api/web-rca/v1/openapi","request_remote_ip":"10.131.0.40:41322"}
      I1017 01:42:04.572384 1 logger.go:101] [opid=34AnchOasnQI5xGVwS2YMMTheWJ] {"response_status":200,"elapsed":"2.59421ms"}
      I1017 01:42:46.891701 1 logger.go:101] SlackChannelCloser archiving ITN-2025-00202 channel: C09B2JUA0AK
      I1017 01:42:46.936482 1 logger.go:101] SlackChannelCloser archiving error: OCM-RCA-31: Cannot perform Slack actions. Slack bot not associated with incident channel
      I1017 01:43:39.210624 1 logger.go:101] [opid=34AnoZiOAZY8N7giBR66znvVAq9] {"request_method":"POST","request_url":"/api/web-rca/v1/slack/events-endpoint","request_remote_ip":"10.129.0.62:49720"}
      I1017 01:43:39.211298 1 logger.go:101] [opid=34AnoZiOAZY8N7giBR66znvVAq9] Handling event type=event_callback/message in channel CCX9DB894
      I1017 01:43:39.211394 1 logger.go:101] [opid=34AnoZiOAZY8N7giBR66znvVAq9] {"response_status":200,"elapsed":"740.09µs"}
      I1017 01:43:39.212444 1 logger.go:101] [opid=34AnoZiOAZY8N7giBR66znvVAq9] Handling event type=event_callback/message in channel CCX9DB894: Cannot get user for SlackId=U03S67J7YSK
      I1017 01:43:46.955853 1 logger.go:101] SlackChannelCloser archiving ITN-2025-00202 channel: C09B2JUA0AK
      I1017 01:43:46.985289 1 logger.go:101] SlackChannelCloser archiving error: OCM-RCA-31: Cannot perform Slack actions. Slack bot not associated with incident channel
      I1017 01:44:29.222758 1 logger.go:101] [opid=34AnuvYcH2DSFkVE2HY87IxPwni] {"request_method":"POST","request_url":"/api/web-rca/v1/slack/events-endpoint","request_remote_ip":"10.129.0.62:34690"}
      I1017 01:44:29.223061 1 logger.go:101] [opid=34AnuvYcH2DSFkVE2HY87IxPwni] Handling event type=event_callback/message in channel CCX9DB894
      I1017 01:44:29.223126 1 logger.go:101] [opid=34AnuvYcH2DSFkVE2HY87IxPwni] {"response_status":200,"elapsed":"337.257µs"}
      I1017 01:44:29.224313 1 logger.go:101] [opid=34AnuvYcH2DSFkVE2HY87IxPwni] Handling event type=event_callback/message in channel CCX9DB894: Cannot get user for SlackId=U01EYSUFX2T
      I1017 01:44:47.003595 1 logger.go:101] SlackChannelCloser archiving ITN-2025-00202 channel: C09B2JUA0AK
      I1017 01:44:47.028492 1 logger.go:101] SlackChannelCloser archiving error: OCM-RCA-31: Cannot perform Slack actions. Slack bot not associated with incident channel
      I1017 01:44:52.384137 1 logger.go:101] [opid=34AnxjbUdWDmwB7r75Q3Pl7Wlnv] {"request_method":"POST","request_url":"/api/web-rca/v1/slack/interactive-endpoint","request_remote_ip":"10.131.0.40:60896"}
      I1017 01:44:52.385978 1 logger.go:101] [opid=34AnxjbUdWDmwB7r75Q3Pl7Wlnv] {"response_status":200,"elapsed":"1.80847ms"}
      panic: runtime error: invalid memory address or nil pointer dereference
      [signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x2143a9c]
      goroutine 3701 [running]:
      gitlab.cee.redhat.com/service/web-rca/pkg/services/domains.updateIncidentWorkingTime({{0xc000aa062b?, 0x2f24768?}
      , {0xc000aa0615?, 0xa0?}}, 0xc00071dc08)
      gitlab.cee.redhat.com/service/web-rca/pkg/services/domains/incident.go:1774 +0x9c
      gitlab.cee.redhat.com/service/web-rca/pkg/services/domains.(*ServiceDomain).UpdateIncident(0xc0006ce008, {0x2f24768, 0xc000b0f830}, 0xc00071dc08, 0xc001366b40)
      gitlab.cee.redhat.com/service/web-rca/pkg/services/domains/incident.go:1426 +0x565
      gitlab.cee.redhat.com/service/web-rca/pkg/slack/modals.(*slackModalIncidentStatusUpdate).Submit(0xc000462b60, {0x2f24768, 0xc000b0f830})
      gitlab.cee.redhat.com/service/web-rca/pkg/slack/modals/incident_status_update_modal.go:141 +0x46f
      gitlab.cee.redhat.com/service/web-rca/pkg/slack/modals.HandleCallback({0x2f24768, 0xc000b0f830}, 0xc000e9be00)
      gitlab.cee.redhat.com/service/web-rca/pkg/slack/modals/slack_modal.go:522 +0x36a
      gitlab.cee.redhat.com/service/web-rca/pkg/handlers.slackHandler.Interactive.func1({
      {0xc00116d170, 0xf}
      , {0xc00069e7f8, 0x18}, {0x0, 0x0}, {0x0, 0x0}, {0xc0006c7040, 0x39}, ...})
      gitlab.cee.redhat.com/service/web-rca/pkg/handlers/slack.go:272 +0x91c
      created by gitlab.cee.redhat.com/service/web-rca/pkg/handlers.slackHandler.Interactive in goroutine 3692
      gitlab.cee.redhat.com/service/web-rca/pkg/handlers/slack.go:215 +0x3d1
      

       

              rhn-support-cmitchel Chris Mitchell
              sraghura Shruthi Raghuraman
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

                Created:
                Updated:
                Resolved: