-
Bug
-
Resolution: Done
-
Undefined
-
Logging 5.5.0
-
None
-
False
-
None
-
False
-
Logging
-
NEW
-
VERIFIED
-
Logging (LogExp) - Sprint 216, Logging (LogExp) - Sprint 218, Logging (LogExp) - Sprint 219, Logging (LogExp) - Sprint 220
Description of problem:
The write index is removed when `the size of the index` > `diskThresholdPercent% * total size`, and the index management job fails with below error:
$ oc logs elasticsearch-im-app-27481395-mlqf5 ======================== Index management delete process starting for app indices = [{"app-000006": {"settings": {"index": {"creation_date": "1648881917870"}}}, "app-000004": {"settings": {"index": {"creation_date": "1648880112384"}}}, "app-000005": {"settings": {"index": {"creation_date": "1648881016385"}}}, "app-000003": {"settings": {"index": {"creation_date": "1648879212096"}}}}] Index app-000006 deleted Index app-000004 deleted Index app-000003 deleted True No indices to delete ======================== Index management rollover process starting for app Current write index for app-write: /tmp/scripts/indexManagementClient.py:97: ElasticsearchDeprecationWarning: [types removal] The parameter include_type_name should be explicitly specified in rollover requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means requests must omit the type name in mapping definitions. response = es_client.indices.rollover(alias=alias, body=decoded) Checking results from _rollover call Calculating next write index based on current write index... Next write index for app-write: -000001 Checking if -000001 exists /tmp/scripts/indexManagementClient.py:109: ElasticsearchDeprecationWarning: [types removal] The parameter include_type_name should be explicitly specified in get indices requests to prepare for 7.0. In 7.0 include_type_name will default to 'false', which means responses will omit the type name in mapping definitions. return es_client.indices.exists(index=index) cat: /tmp/response.txt: No such file or directory
Before the job running, the indices look like:
$ oc exec elasticsearch-cdm-bz12o5mh-1-5fc5b69f8f-ttrhg -- es_util --query=_cat/indices?v |grep app
Defaulted container "elasticsearch" out of: elasticsearch, proxy
green open app-000005 9aULOT9iRoq0PxY-TUCTKQ 3 1 1664508 0 2.6gb 1.3gb
green open app-000003 ChZCDRaEQDOP7sy0qYApew 3 1 1664142 0 2.6gb 1.3gb
green open app-000006 oOOW9pObQdeV5ABAIUJgag 3 1 3213237 0 6.2gb 3.1gb
green open app-000004 FhhCUaImSkurVv5IhT566g 3 1 1665241 0 2.6gb 1.3gb
after:
$ oc exec elasticsearch-cdm-bz12o5mh-1-5fc5b69f8f-ttrhg -- es_util --query=_cat/indices?v |grep app
Defaulted container "elasticsearch" out of: elasticsearch, proxy
green open app-000005 9aULOT9iRoq0PxY-TUCTKQ 3 1 1664508 0 2.6gb 1.3gb
ES configurations:
indexManagement:
mappings:
- aliases:
- app
- logs.app
name: app
policyRef: app-policy
- aliases:
- infra
- logs.infra
name: infra
policyRef: infra-policy
- aliases:
- audit
- logs.audit
name: audit
policyRef: audit-policy
policies:
- name: app-policy
phases:
delete:
diskThresholdPercent: 5
minAge: 12h
pruneNamespacesInterval: 30m
hot:
actions:
rollover:
maxAge: 15m
pollInterval: 15m
- name: infra-policy
phases:
delete:
diskThresholdPercent: 30
minAge: 12h
pruneNamespacesInterval: 30m
hot:
actions:
rollover:
maxAge: 15m
pollInterval: 15m
- name: audit-policy
phases:
delete:
minAge: 1d
pruneNamespacesInterval: 30m
hot:
actions:
rollover:
maxAge: 1h
pollInterval: 15m
managementState: Managed
nodeSpec:
proxyResources:
limits:
memory: 256Mi
requests:
cpu: 100m
memory: 256Mi
resources:
requests:
memory: 2Gi
nodes:
- genUUID: bz12o5mh
nodeCount: 3
proxyResources: {}
resources: {}
roles:
- client
- data
- master
storage:
size: 20Gi
storageClassName: gp2
redundancyPolicy: SingleRedundancy
Version-Release number of selected component (if applicable):
EO image: quay.io/openshift-logging/elasticsearch-operator@sha256:205178d9cc1229ad9393d94b2f769fa124ccd5cad373f3bcb7afc6726a64d2e4
How reproducible:
Always
Steps to Reproduce:
1. deploy logging, set logStore.elasticsearch.storage.size to 20Gi
2. enable percentage based index management, set `diskThresholdPercent: 5` for application logs(this need to set CL/instance to Unmanaged)
3. create some apps to generate logs, if the size of the write index > 1Gi, when the index management job running, the write index will be removed and no write index will be created.
Actual results:
Expected results:
Additional info:
