Uploaded image for project: 'Red Hat Fuse'
  1. Red Hat Fuse
  2. ENTESB-17796

CSV parameters are not updated in AtlasMap step

    XMLWordPrintable

Details

    • Bug
    • Resolution: Done
    • Critical
    • fuse-7.11-GA
    • fuse-7.10-GA
    • Fuse Online
    • None
    • False
    • False
    • % %
    • Todo
    • Hide

      When you edit CSV parameters in data-shape definition in some step, you need to add something (e.g. unused constant) into DataMapper in order to the atlasmapping is updated

      Show
      When you edit CSV parameters in data-shape definition in some step, you need to add something (e.g. unused constant) into DataMapper in order to the atlasmapping is updated

    Description

      I have created and published an integration (Webhook -> DB) which consumes CSV. I used this CSV parameters in the WebHook step: allowDuplicateHeaderNames=true, firstRecordAsHeader=true.
      integrationWithCsv-export-firstVersion.zip
      After that, I edited the webhook step, changed CSV parameters (allowDuplicateHeaderNames=false, added ignotreEmptyLines=true) and redeploy but the functionality didn't changed.
      integrationWithCsv-export-afterEditWebhook.zip
      When I compare the model.json files, the parameters are updated in the WebHook step, but not in the AtlasMap step.
      First version:

      ...
"parameters":{
  "allowDuplicateHeaderNames":"true",
  "firstRecordAsHeader":"true",
  "format":"Default"
},
"specification":"sourceCsvString,sourceCsvNumber,sourceCsvDecimal,sourceCsvDate,sourceCsvBoolean\ncsv0,0,0.0,1989-05-05,true\ncsv1,10,10.0,1989-05-05,true"
},
...
"atlasmapping":"{\"AtlasMapping\":{\"jsonType\":\"io.atlasmap.v2.AtlasMapping\",\"dataSource\":[{\"jsonType\":\"io.atlasmap.v2.DataSource\",\"id\":\"-Mo9A76867eZXx4EWHvx\",\"name\":\"1 - undefined\",\"description\":\"-Mo9A76867eZXx4EWHvx\",\"uri\":\"atlas:csv:-Mo9A76867eZXx4EWHvx?format=Default&allowDuplicateHeaderNames=true&firstRecordAsHeader=true\",\"dataSourceType\":\"SOURCE\"},{\"jsonType\":\"io.atlasmap.json.v2.JsonDataSource\",\"id\":\"-Mo9AAgx67eZXx4EWHvx\",\"name\":\"3 - SQL Parameter\",\"description\":\"Parameters of SQL [insert into todo (task) values (:#val)]\",\"uri\":\"atlas:json:-Mo9AAgx67eZXx4EWHvx\",\"dataSourceType\":\"TARGET\"}],\"mappings\":{\"mapping\":[{\"jsonType\":\"io.atlasmap.v2.Mapping\",\"id\":\"mapping.338231\",\"inputFieldGroup\":{\"jsonType\":\"io.atlasmap.v2.FieldGroup\",\"actions\":[{\"delimiter\":\" \",\"delimitingEmptyValues\":\"true\",\"@type\":\"Concatenate\"}],\"field\":[{\"jsonType\":\"io.atlasmap.csv.v2.CsvField\",\"name\":\"sourceCsvString\",\"path\":\"/<>/sourceCsvString\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9A76867eZXx4EWHvx\"}]},\"outputField\":[{\"jsonType\":\"io.atlasmap.json.v2.JsonField\",\"name\":\"val\",\"path\":\"/val\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9AAgx67eZXx4EWHvx\"}]}]},\"name\":\"UI.0\",\"lookupTables\":{\"lookupTable\":[]},\"constants\":{\"constant\":[]},\"properties\":{\"property\":[]}}}"

      After edit webhook step:

      ...
"parameters":{
  "allowDuplicateHeaderNames":"false",
  "firstRecordAsHeader":"true",
  "format":"Default",
  "ignoreHeaderCase":"true"
},                "specification":"sourceCsvString,sourceCsvNumber,sourceCsvDecimal,sourceCsvDate,sourceCsvBoolean\ncsv0,0,0.0,1989-05-05,true\ncsv1,10,10.0,1989-05-05,true"
},
...
"atlasmapping":"{\"AtlasMapping\":{\"jsonType\":\"io.atlasmap.v2.AtlasMapping\",\"dataSource\":[{\"jsonType\":\"io.atlasmap.v2.DataSource\",\"id\":\"-Mo9A76867eZXx4EWHvx\",\"name\":\"1 - undefined\",\"description\":\"-Mo9A76867eZXx4EWHvx\",\"uri\":\"atlas:csv:-Mo9A76867eZXx4EWHvx?format=Default&allowDuplicateHeaderNames=true&firstRecordAsHeader=true\",\"dataSourceType\":\"SOURCE\"},{\"jsonType\":\"io.atlasmap.json.v2.JsonDataSource\",\"id\":\"-Mo9AAgx67eZXx4EWHvx\",\"name\":\"3 - SQL Parameter\",\"description\":\"Parameters of SQL [insert into todo (task) values (:#val)]\",\"uri\":\"atlas:json:-Mo9AAgx67eZXx4EWHvx\",\"dataSourceType\":\"TARGET\"}],\"mappings\":{\"mapping\":[{\"jsonType\":\"io.atlasmap.v2.Mapping\",\"id\":\"mapping.338231\",\"inputFieldGroup\":{\"jsonType\":\"io.atlasmap.v2.FieldGroup\",\"actions\":[{\"delimiter\":\" \",\"delimitingEmptyValues\":\"true\",\"@type\":\"Concatenate\"}],\"field\":[{\"jsonType\":\"io.atlasmap.csv.v2.CsvField\",\"name\":\"sourceCsvString\",\"path\":\"/<>/sourceCsvString\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9A76867eZXx4EWHvx\"}]},\"outputField\":[{\"jsonType\":\"io.atlasmap.json.v2.JsonField\",\"name\":\"val\",\"path\":\"/val\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9AAgx67eZXx4EWHvx\"}]}]},\"name\":\"UI.0\",\"lookupTables\":{\"lookupTable\":[]},\"constants\":{\"constant\":[]},\"properties\":{\"property\":[]}}}"

      Also, when I open DataMapper step and save the integration, the atlasmapping is not updated. I have to do some changes in Data Mapper (e.g. create a dummy constant).
      integrationWithCsv-export-afterAddConstant.zip

      ...
"parameters":{
  "allowDuplicateHeaderNames":"false",
  "firstRecordAsHeader":"true",
  "format":"Default",
  "ignoreHeaderCase":"true"
},                "specification":"sourceCsvString,sourceCsvNumber,sourceCsvDecimal,sourceCsvDate,sourceCsvBoolean\ncsv0,0,0.0,1989-05-05,true\ncsv1,10,10.0,1989-05-05,true"
},
...
"atlasmapping":"{\"AtlasMapping\":{\"jsonType\":\"io.atlasmap.v2.AtlasMapping\",\"dataSource\":[{\"jsonType\":\"io.atlasmap.v2.DataSource\",\"id\":\"-Mo9A76867eZXx4EWHvx\",\"name\":\"1 - undefined\",\"description\":\"-Mo9A76867eZXx4EWHvx\",\"uri\":\"atlas:csv:-Mo9A76867eZXx4EWHvx?format=Default&allowDuplicateHeaderNames=false&firstRecordAsHeader=true&ignoreHeaderCase=true\",\"dataSourceType\":\"SOURCE\"},{\"jsonType\":\"io.atlasmap.json.v2.JsonDataSource\",\"id\":\"-Mo9AAgx67eZXx4EWHvx\",\"name\":\"4 - SQL Parameter\",\"description\":\"Parameters of SQL [insert into todo (task) values (:#val)]\",\"uri\":\"atlas:json:-Mo9AAgx67eZXx4EWHvx\",\"dataSourceType\":\"TARGET\"}],\"mappings\":{\"mapping\":[{\"jsonType\":\"io.atlasmap.v2.Mapping\",\"id\":\"mapping.338231\",\"inputFieldGroup\":{\"jsonType\":\"io.atlasmap.v2.FieldGroup\",\"actions\":[{\"delimiter\":\" \",\"delimitingEmptyValues\":\"true\",\"@type\":\"Concatenate\"}],\"field\":[{\"jsonType\":\"io.atlasmap.csv.v2.CsvField\",\"name\":\"sourceCsvString\",\"path\":\"/<>/sourceCsvString\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9A76867eZXx4EWHvx\"}]},\"outputField\":[{\"jsonType\":\"io.atlasmap.json.v2.JsonField\",\"name\":\"val\",\"path\":\"/val\",\"fieldType\":\"STRING\",\"docId\":\"-Mo9AAgx67eZXx4EWHvx\"}]}]},\"name\":\"UI.0\",\"lookupTables\":{\"lookupTable\":[]},\"constants\":{\"constant\":[{\"name\":\"dummy\",\"value\":\"dummy\",\"fieldType\":\"STRING\"}]},\"properties\":{\"property\":[]}}}"
                     },

      In my POV, the user should open the DataMapper after they edited the CSV parameters in the WebHook step to check that mapping is still valid, but after that, the mapping should update without any changes.

      Attachments

        Issue Links

          is caused by

          Enhancement - An enhancement to or refactoring of existing functionality that is not configurable by an end user (typically a change made by an internal team that affects users) ENTESB-15928 CSV support in Fuse Online

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Done
          is documented by

          Task - Represents a small unit of work that is not end-user facing. FUSEDOC-4853 Document Data Mapper CSV support requires editing step

          • Major - To be worked after higher priority work items are addressed. When the severity is high and the effort to change it is low to moderate, this term is chosen. Issues in this category likely have an existing workaround but implementation or execution may be non-trivial.
          • Verified
          links to

          Web Link https://github.com/atlasmap/atlasmap/issues/2990

          Activity

            People

              zregvart@redhat.com Zoran Regvart
              mkralik@redhat.com Matej Kralik
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: