Major We are seeing these errors from the pods in disruption-uploader and alert-uploader:
$ oc logs disruption-uploader-28517040-bkzj5 time="2024-03-21T12:11:25.235Z" level=info msg="Locating jobs" time="2024-03-21T12:11:26.672Z" level=info msg="got last uploaded job run end time" lastUploadedJobRun="2024-03-19 07:59:14 +0000 UTC" time="2024-03-21T12:11:26.672Z" level=info msg="listing prow jobs since" since="2024-03-19 07:29:14 +0000 UTC" time="2024-03-21T12:11:27.089Z" level=info msg="found existing job run IDs" idCount=26 time="2024-03-21T12:11:32.341Z" level=fatal msg="Command failed" error="error listing job runs to import: bigquery: NULL cannot be assigned to field `StartTime` of type Time" $ oc logs alert-uploader-28517040-szq2f time="2024-03-21T12:03:11.943Z" level=info msg="loaded cache of known alerts across all releases" knownAlerts=2438 time="2024-03-21T12:03:11.943Z" level=info msg="Locating jobs" time="2024-03-21T12:03:12.931Z" level=info msg="got last uploaded job run end time" lastUploadedJobRun="2024-03-19 07:59:14 +0000 UTC" time="2024-03-21T12:03:12.931Z" level=info msg="listing prow jobs since" since="2024-03-19 07:29:14 +0000 UTC" time="2024-03-21T12:03:14.068Z" level=info msg="found existing job run IDs" idCount=26 time="2024-03-21T12:03:19.356Z" level=fatal msg="Command failed" error="error listing job runs to import: bigquery: NULL cannot be assigned to field `StartTime` of type Time"
I think it's from https://github.com/openshift/ci-tools/blob/2ccf98edeb22509ee82f535d6c70861f3b18a528/pkg/jobrunaggregator/jobrunaggregatorlib/ci_data_client.go#L348-L350
This query:
SELECT prowjob_url, prowjob_build_id FROM `openshift-gce-devel.ci_analysis_us.jobs`
WHERE
created >= '2024-03-18'
AND prowjob_start IS NULL
LIMIT 10
returns a job that has a null prowjob_start:
https://prow.ci.openshift.org/view/gs/test-platform-results/logs/foobar/1770035372357586944
