Uploaded image for project: 'Infinispan'
  1. Infinispan
  2. ISPN-4022

M/R: Run the combiner concurrently with the mapper

    Details

      Description

      Because we only run the combiner after we finished the mapping phase, we need to keep all the results of the mapping phase in memory at once. We should split the output of the mapper into chunks and allow the combiner to process chunks while the mapper is still running, relieving some of the memory pressure. Maybe even block the mapper if there are too many chunks in-flight.

        Gliffy Diagrams

          Attachments

            Issue Links

              Activity

                People

                • Assignee:
                  vblagojevic Vladimir Blagojevic
                  Reporter:
                  dan.berindei Dan Berindei
                • Votes:
                  1 Vote for this issue
                  Watchers:
                  2 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved: