-
Bug
-
Resolution: Done
-
Blocker
-
Logging 5.6.0
-
False
-
None
-
False
-
NEW
-
OBSDA-108 - Distribute an alternate Vector Log Collector
-
VERIFIED
-
-
Log Collection - Sprint 226, Log Collection - Sprint 227
Description of problem:
When using vector as collector, the logs can't be forwarded to non-clusterlogging managed ES. Many errors in vector pod:
2022-10-25T01:23:15.640676Z ERROR sink{component_kind="sink" component_id=es_created_by_user component_type=elasticsearch component_name=es_created_by_user}:request{request_id=0}: vector::internal_events::elasticsearch: Response failed. error_code=http_response_400 error_type="request_failed" stage="sending" response=Response { status: 400, version: HTTP/1.1, headers: {"content-type": "application/json; charset=UTF-8", "content-length": "11527"}, body: b"{\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;27: type is missing;28: type is missing;29: type is missing;30: type is missing;31: type is missing;32: type is missing;33: type is missing;34: type is missing;35: type is missing;36: type is missing;37: type is missing;38: type is missing;39: type is missing;40: type is missing;41: type is missing;42: type is missing;43: type is missing;44: type is missing;45: type is missing;46: type is missing;47: type is missing;48: type is missing;49: type is missing;50: type is missing;51: type is missing;52: type is missing;53: type is missing;54: type is missing;55: type is missing;56: type is missing;57: type is missing;58: type is missing;59: type is missing;60: type is missing;61: type is missing;62: type is missing;63: type is missing;64: type is missing;65: type is missing;66: type is missing;67: type is missing;68: type is missing;69: type is missing;70: type is missing;71: type is missing;72: type is missing;73: type is missing;74: type is missing;75: type is missing;76: type is missing;77: type is missing;78: type is missing;79: type is missing;80: type is missing;81: type is missing;82: type is missing;83: type is missing;84: type is missing;85: type is missing;86: type is missing;87: type is missing;88: type is missing;89: type is missing;90: type is missing;91: type is missing;92: type is missing;93: type is missing;94: type is missing;95: type is missing;96: type is missing;97: type is missing;98: type is missing;99: type is missing;100: type is missing;101: type is missing;102: type is missing;103: type is missing;104: type is missing;105: type is missing;106: type is missing;107: type is missing;108: type is missing;109: type is missing;110: type is missing;111: type is missing;112: type is missing;113: type is missing;114: type is missing;115: type is missing;116: type is missing;117: type is missing;118: type is missing;119: type is missing;120: type is missing;121: type is missing;122: type is missing;123: type is missing;124: type is missing;125: type is missing;126: type is missing;127: type is missing;128: type is missing;129: type is missing;130: type is missing;131: type is missing;132: type is missing;133: type is missing;134: type is missing;135: type is missing;136: type is missing;137: type is missing;138: type is missing;139: type is missing;140: type is missing;141: type is missing;142: type is missing;143: type is missing;144: type is missing;145: type is missing;146: type is missing;147: type is missing;148: type is missing;149: type is missing;150: type is missing;151: type is missing;152: type is missing;153: type is missing;154: type is missing;155: type is missing;156: type is missing;157: type is missing;158: type is missing;159: type is missing;160: type is missing;161: type is missing;162: type is missing;163: type is missing;164: type is missing;165: type is missing;166: type is missing;167: type is missing;168: type is missing;169: type is missing;170: type is missing;171: type is missing;172: type is missing;173: type is missing;174: type is missing;175: type is missing;176: type is missing;177: type is missing;178: type is missing;179: type is missing;180: type is missing;181: type is missing;182: type is missing;183: type is missing;184: type is missing;185: type is missing;186: type is missing;187: type is missing;188: type is missing;189: type is missing;190: type is missing;191: type is missing;192: type is missing;193: type is missing;194: type is missing;195: type is missing;196: type is missing;197: type is missing;198: type is missing;199: type is missing;200: type is missing;201: type is missing;202: type is missing;203: type is missing;204: type is missing;205: type is missing;206: type is missing;207: type is missing;208: type is missing;209: type is missing;210: type is missing;211: type is missing;212: type is missing;213: type is missing;214: type is missing;215: type is missing;216: type is missing;217: type is missing;218: type is missing;219: type is missing;220: type is missing;221: type is missing;222: type is missing;223: type is missing;224: type is missing;225: type is missing;226: type is missing;227: type is missing;228: type is missing;229: type is missing;230: type is missing;231: type is missing;232: type is missing;233: type is missing;234: type is missing;235: type is missing;236: type is missing;237: type is missing;238: type is missing;239: type is missing;240: type is missing;241: type is missing;242: type is missing;243: type is missing;244: type is missing;245: type is missing;246: type is missing;247: type is missing;248: type is missing;249: type is missing;250: type is missing;251: type is missing;252: type is missing;253: type is missing;254: type is missing;255: type is missing;256: type is missing;257: type is missing;258: type is missing;259: type is missing;260: type is missing;261: type is missing;262: type is missing;263: type is missing;264: type is missing;265: type is missing;266: type is missing;267: type is missing;268: type is missing;269: type is missing;270: type is missing;271: type is missing;272: type is missing;273: type is missing;274: type is missing;275: type is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;27: type is missing;28: type is missing;29: type is missing;30: type is missing;31: type is missing;32: type is missing;33: type is missing;34: type is missing;35: type is missing;36: type is missing;37: type is missing;38: type is missing;39: type is missing;40: type is missing;41: type is missing;42: type is missing;43: type is missing;44: type is missing;45: type is missing;46: type is missing;47: type is missing;48: type is missing;49: type is missing;50: type is missing;51: type is missing;52: type is missing;53: type is missing;54: type is missing;55: type is missing;56: type is missing;57: type is missing;58: type is missing;59: type is missing;60: type is missing;61: type is missing;62: type is missing;63: type is missing;64: type is missing;65: type is missing;66: type is missing;67: type is missing;68: type is missing;69: type is missing;70: type is missing;71: type is missing;72: type is missing;73: type is missing;74: type is missing;75: type is missing;76: type is missing;77: type is missing;78: type is missing;79: type is missing;80: type is missing;81: type is missing;82: type is missing;83: type is missing;84: type is missing;85: type is missing;86: type is missing;87: type is missing;88: type is missing;89: type is missing;90: type is missing;91: type is missing;92: type is missing;93: type is missing;94: type is missing;95: type is missing;96: type is missing;97: type is missing;98: type is missing;99: type is missing;100: type is missing;101: type is missing;102: type is missing;103: type is missing;104: type is missing;105: type is missing;106: type is missing;107: type is missing;108: type is missing;109: type is missing;110: type is missing;111: type is missing;112: type is missing;113: type is missing;114: type is missing;115: type is missing;116: type is missing;117: type is missing;118: type is missing;119: type is missing;120: type is missing;121: type is missing;122: type is missing;123: type is missing;124: type is missing;125: type is missing;126: type is missing;127: type is missing;128: type is missing;129: type is missing;130: type is missing;131: type is missing;132: type is missing;133: type is missing;134: type is missing;135: type is missing;136: type is missing;137: type is missing;138: type is missing;139: type is missing;140: type is missing;141: type is missing;142: type is missing;143: type is missing;144: type is missing;145: type is missing;146: type is missing;147: type is missing;148: type is missing;149: type is missing;150: type is missing;151: type is missing;152: type is missing;153: type is missing;154: type is missing;155: type is missing;156: type is missing;157: type is missing;158: type is missing;159: type is missing;160: type is missing;161: type is missing;162: type is missing;163: type is missing;164: type is missing;165: type is missing;166: type is missing;167: type is missing;168: type is missing;169: type is missing;170: type is missing;171: type is missing;172: type is missing;173: type is missing;174: type is missing;175: type is missing;176: type is missing;177: type is missing;178: type is missing;179: type is missing;180: type is missing;181: type is missing;182: type is missing;183: type is missing;184: type is missing;185: type is missing;186: type is missing;187: type is missing;188: type is missing;189: type is missing;190: type is missing;191: type is missing;192: type is missing;193: type is missing;194: type is missing;195: type is missing;196: type is missing;197: type is missing;198: type is missing;199: type is missing;200: type is missing;201: type is missing;202: type is missing;203: type is missing;204: type is missing;205: type is missing;206: type is missing;207: type is missing;208: type is missing;209: type is missing;210: type is missing;211: type is missing;212: type is missing;213: type is missing;214: type is missing;215: type is missing;216: type is missing;217: type is missing;218: type is missing;219: type is missing;220: type is missing;221: type is missing;222: type is missing;223: type is missing;224: type is missing;225: type is missing;226: type is missing;227: type is missing;228: type is missing;229: type is missing;230: type is missing;231: type is missing;232: type is missing;233: type is missing;234: type is missing;235: type is missing;236: type is missing;237: type is missing;238: type is missing;239: type is missing;240: type is missing;241: type is missing;242: type is missing;243: type is missing;244: type is missing;245: type is missing;246: type is missing;247: type is missing;248: type is missing;249: type is missing;250: type is missing;251: type is missing;252: type is missing;253: type is missing;254: type is missing;255: type is missing;256: type is missing;257: type is missing;258: type is missing;259: type is missing;260: type is missing;261: type is missing;262: type is missing;263: type is missing;264: type is missing;265: type is missing;266: type is missing;267: type is missing;268: type is missing;269: type is missing;270: type is missing;271: type is missing;272: type is missing;273: type is missing;274: type is missing;275: type is missing;\"},\"status\":400}" } 2022-10-25T01:23:15.640900Z ERROR sink{component_kind="sink" component_id=es_created_by_user component_type=elasticsearch component_name=es_created_by_user}:request{request_id=0}: vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;27: type is missing;28: type is missing;29: type is missing;30: type is missing;31: type is missing;32: type is missing;33: type is missing;34: type is missing;35: type is missing;36: type is missing;37: type is missing;38: type is missing;39: type is missing;40: type is missing;41: type is missing;42: type is missing;43: type is missing;44: type is missing;45: type is missing;46: type is missing;47: type is missing;48: type is missing;49: type is missing;50: type is missing;51: type is missing;52: type is missing;53: type is missing;54: type is missing;55: type is missing;56: type is missing;57: type is missing;58: type is missing;59: type is missing;60: type is missing;61: type is missing;62: type is missing;63: type is missing;64: type is missing;65: type is missing;66: type is missing;67: type is missing;68: type is missing;69: type is missing;70: type is missing;71: type is missing;72: type is missing;73: type is missing;74: type is missing;75: type is missing;76: type is missing;77: type is missing;78: type is missing;79: type is missing;80: type is missing;81: type is missing;82: type is missing;83: type is missing;84: type is missing;85: type is missing;86: type is missing;87: type is missing;88: type is missing;89: type is missing;90: type is missing;91: type is missing;92: type is missing;93: type is missing;94: type is missing;95: type is missing;96: type is missing;97: type is missing;98: type is missing;99: type is missing;100: type is missing;101: type is missing;102: type is missing;103: type is missing;104: type is missing;105: type is missing;106: type is missing;107: type is missing;108: type is missing;109: type is missing;110: type is missing;111: type is missing;112: type is missing;113: type is missing;114: type is missing;115: type is missing;116: type is missing;117: type is missing;118: type is missing;119: type is missing;120: type is missing;121: type is missing;122: type is missing;123: type is missing;124: type is missing;125: type is missing;126: type is missing;127: type is missing;128: type is missing;129: type is missing;130: type is missing;131: type is missing;132: type is missing;133: type is missing;134: type is missing;135: type is missing;136: type is missing;137: type is missing;138: type is missing;139: type is missing;140: type is missing;141: type is missing;142: type is missing;143: type is missing;144: type is missing;145: type is missing;146: type is missing;147: type is missing;148: type is missing;149: type is missing;150: type is missing;151: type is missing;152: type is missing;153: type is missing;154: type is missing;155: type is missing;156: type is missing;157: type is missing;158: type is missing;159: type is missing;160: type is missing;161: type is missing;162: type is missing;163: type is missing;164: type is missing;165: type is missing;166: type is missing;167: type is missing;168: type is missing;169: type is missing;170: type is missing;171: type is missing;172: type is missing;173: type is missing;174: type is missing;175: type is missing;176: type is missing;177: type is missing;178: type is missing;179: type is missing;180: type is missing;181: type is missing;182: type is missing;183: type is missing;184: type is missing;185: type is missing;186: type is missing;187: type is missing;188: type is missing;189: type is missing;190: type is missing;191: type is missing;192: type is missing;193: type is missing;194: type is missing;195: type is missing;196: type is missing;197: type is missing;198: type is missing;199: type is missing;200: type is missing;201: type is missing;202: type is missing;203: type is missing;204: type is missing;205: type is missing;206: type is missing;207: type is missing;208: type is missing;209: type is missing;210: type is missing;211: type is missing;212: type is missing;213: type is missing;214: type is missing;215: type is missing;216: type is missing;217: type is missing;218: type is missing;219: type is missing;220: type is missing;221: type is missing;222: type is missing;223: type is missing;224: type is missing;225: type is missing;226: type is missing;227: type is missing;228: type is missing;229: type is missing;230: type is missing;231: type is missing;232: type is missing;233: type is missing;234: type is missing;235: type is missing;236: type is missing;237: type is missing;238: type is missing;239: type is missing;240: type is missing;241: type is missing;242: type is missing;243: type is missing;244: type is missing;245: type is missing;246: type is missing;247: type is missing;248: type is missing;249: type is missing;250: type is missing;251: type is missing;252: type is missing;253: type is missing;254: type is missing;255: type is missing;256: type is missing;257: type is missing;258: type is missing;259: type is missing;260: type is missing;261: type is missing;262: type is missing;263: type is missing;264: type is missing;265: type is missing;266: type is missing;267: type is missing;268: type is missing;269: type is missing;270: type is missing;271: type is missing;272: type is missing;273: type is missing;274: type is missing;275: type is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;22: type is missing;23: type is missing;24: type is missing;25: type is missing;26: type is missing;27: type is missing;28: type is missing;29: type is missing;30: type is missing;31: type is missing;32: type is missing;33: type is missing;34: type is missing;35: type is missing;36: type is missing;37: type is missing;38: type is missing;39: type is missing;40: type is missing;41: type is missing;42: type is missing;43: type is missing;44: type is missing;45: type is missing;46: type is missing;47: type is missing;48: type is missing;49: type is missing;50: type is missing;51: type is missing;52: type is missing;53: type is missing;54: type is missing;55: type is missing;56: type is missing;57: type is missing;58: type is missing;59: type is missing;60: type is missing;61: type is missing;62: type is missing;63: type is missing;64: type is missing;65: type is missing;66: type is missing;67: type is missing;68: type is missing;69: type is missing;70: type is missing;71: type is missing;72: type is missing;73: type is missing;74: type is missing;75: type is missing;76: type is missing;77: type is missing;78: type is missing;79: type is missing;80: type is missing;81: type is missing;82: type is missing;83: type is missing;84: type is missing;85: type is missing;86: type is missing;87: type is missing;88: type is missing;89: type is missing;90: type is missing;91: type is missing;92: type is missing;93: type is missing;94: type is missing;95: type is missing;96: type is missing;97: type is missing;98: type is missing;99: type is missing;100: type is missing;101: type is missing;102: type is missing;103: type is missing;104: type is missing;105: type is missing;106: type is missing;107: type is missing;108: type is missing;109: type is missing;110: type is missing;111: type is missing;112: type is missing;113: type is missing;114: type is missing;115: type is missing;116: type is missing;117: type is missing;118: type is missing;119: type is missing;120: type is missing;121: type is missing;122: type is missing;123: type is missing;124: type is missing;125: type is missing;126: type is missing;127: type is missing;128: type is missing;129: type is missing;130: type is missing;131: type is missing;132: type is missing;133: type is missing;134: type is missing;135: type is missing;136: type is missing;137: type is missing;138: type is missing;139: type is missing;140: type is missing;141: type is missing;142: type is missing;143: type is missing;144: type is missing;145: type is missing;146: type is missing;147: type is missing;148: type is missing;149: type is missing;150: type is missing;151: type is missing;152: type is missing;153: type is missing;154: type is missing;155: type is missing;156: type is missing;157: type is missing;158: type is missing;159: type is missing;160: type is missing;161: type is missing;162: type is missing;163: type is missing;164: type is missing;165: type is missing;166: type is missing;167: type is missing;168: type is missing;169: type is missing;170: type is missing;171: type is missing;172: type is missing;173: type is missing;174: type is missing;175: type is missing;176: type is missing;177: type is missing;178: type is missing;179: type is missing;180: type is missing;181: type is missing;182: type is missing;183: type is missing;184: type is missing;185: type is missing;186: type is missing;187: type is missing;188: type is missing;189: type is missing;190: type is missing;191: type is missing;192: type is missing;193: type is missing;194: type is missing;195: type is missing;196: type is missing;197: type is missing;198: type is missing;199: type is missing;200: type is missing;201: type is missing;202: type is missing;203: type is missing;204: type is missing;205: type is missing;206: type is missing;207: type is missing;208: type is missing;209: type is missing;210: type is missing;211: type is missing;212: type is missing;213: type is missing;214: type is missing;215: type is missing;216: type is missing;217: type is missing;218: type is missing;219: type is missing;220: type is missing;221: type is missing;222: type is missing;223: type is missing;224: type is missing;225: type is missing;226: type is missing;227: type is missing;228: type is missing;229: type is missing;230: type is missing;231: type is missing;232: type is missing;233: type is missing;234: type is missing;235: type is missing;236: type is missing;237: type is missing;238: type is missing;239: type is missing;240: type is missing;241: type is missing;242: type is missing;243: type is missing;244: type is missing;245: type is missing;246: type is missing;247: type is missing;248: type is missing;249: type is missing;250: type is missing;251: type is missing;252: type is missing;253: type is missing;254: type is missing;255: type is missing;256: type is missing;257: type is missing;258: type is missing;259: type is missing;260: type is missing;261: type is missing;262: type is missing;263: type is missing;264: type is missing;265: type is missing;266: type is missing;267: type is missing;268: type is missing;269: type is missing;270: type is missing;271: type is missing;272: type is missing;273: type is missing;274: type is missing;275: type is missing;\"},\"status\":400}" 2022-10-25T01:23:16.822805Z ERROR sink{component_kind="sink" component_id=es_created_by_user component_type=elasticsearch component_name=es_created_by_user}:request{request_id=1}: vector::internal_events::elasticsearch: Response failed. error_code=http_response_400 error_type="request_failed" stage="sending" response=Response { status: 400, version: HTTP/1.1, headers: {"content-type": "application/json; charset=UTF-8", "content-length": "1015"}, body: b"{\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;\"},\"status\":400}" } 2022-10-25T01:23:16.822859Z ERROR sink{component_kind="sink" component_id=es_created_by_user component_type=elasticsearch component_name=es_created_by_user}:request{request_id=1}: vector::sinks::util::retries: Not retriable; dropping the request. reason="client-side error, 400 Bad Request: {\"error\":{\"root_cause\":[{\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;\"}],\"type\":\"action_request_validation_exception\",\"reason\":\"Validation Failed: 1: type is missing;2: type is missing;3: type is missing;4: type is missing;5: type is missing;6: type is missing;7: type is missing;8: type is missing;9: type is missing;10: type is missing;11: type is missing;12: type is missing;13: type is missing;14: type is missing;15: type is missing;16: type is missing;17: type is missing;18: type is missing;19: type is missing;20: type is missing;21: type is missing;\"},\"status\":400}"
Vector config files: openssl.cnf
$ oc get clf instance -oyaml apiVersion: logging.openshift.io/v1 kind: ClusterLogForwarder metadata: creationTimestamp: "2022-10-25T01:18:15Z" generation: 1 name: instance namespace: openshift-logging resourceVersion: "54532" uid: f9a5f13e-516c-4521-9db0-4edb5a7d37c2 spec: outputs: - name: es-created-by-user type: elasticsearch url: http://elasticsearch-server.e2e-test-vector-es-namespace-f8bfl.svc:9200 pipelines: - inputRefs: - infrastructure - application - audit name: forward-to-external-es outputRefs: - es-created-by-user $ oc get cl instance -oyaml apiVersion: logging.openshift.io/v1 kind: ClusterLogging metadata: generation: 1 name: instance namespace: openshift-logging resourceVersion: "48965" uid: b352b792-91a4-4090-853f-5e1c906ed1cf spec: collection: type: vector managementState: Managed
Version-Release number of selected component (if applicable):
cluster-logging.v5.6.0
How reproducible:
Always
Steps to Reproduce:
- deploy an elasticsearch server
- deploy logging 5.6
- forward logs to elasticsearch server using vector
Actual results:
No logs can be forwarded to elasticsearch server, many errors in vector
Expected results:
No error, logs can be forwarded to elasticsearch server
Additional info:
No issue when using fluentd as collector.
No issue in logging 5.5.
No issue when forward logs to clusterlogging managed ES using vector.