Answers

• kthorngren Posts: 20,309Questions: 26Answers: 4,769

  February 2020 edited July 2020

  I found that the above code works ok for a small number of records. The problem is that it doesn't control the ajax requests so they are sent simultaneously. In my environment this causes timeouts. I changed the way the promises are handled so the ajax requests run sequentially. Here is the code:

  var promises = [Promise.resolve()];
  
  /**********************************************
      Save data to DB functions
  **********************************************/
  
  // Promise ajax function to post data to DB.
  // Parameters:
  // data: CSV data to upload to server
  // count: beginning file number for batch
  // numFiles: Ending file number for batch
  // totalRecords:  Total number of files uploaded
  // moreFiles: boolean indicating if this is the last set of files
  function postDbData(data, count, numFiles, totalRecords, moreFiles) {
  
      // If no more files then use ajaxstop to relaod the Datatable.
      if ( !moreFiles ) {
          $( document ).one().ajaxStop(function() {
  
              // Reload table after all batches are complete
              $(document).off().ajaxStop();
              console.log('Batch upload complete - reloading table')
              table = $('#main-table').DataTable();
              table.ajax.reload( null, false );
              $.unblockUI();
  
              if (errors) {
                  alert('The following errors occurred:\n - ' + errors.join('\n - '));
              }
  
          });
      }
      console.log('postDbData - Uploading records ' + count + ' to ' + numFiles + ' of ' + totalRecords +' records')
      addMsgDiv();
      $("#msg").html('Uploading records ' + count + ' to ' + numFiles + ' of ' + totalRecords +' records', false);
      $.blockUI({ message: $('#msg') });   // Block UI and display status
      return new Promise((resolve, reject) => {
          $.ajax({
              type: "POST",
              url: "/api/v1/locations/create",
              data: {data: data},
              success: function(json) {
                  $.unblockUI();  // Unblock UI
                  resolve();
              },
              error: function() {
                  data = [];
                  $.unblockUI();
                  errors.push('Error uploading ' + count + ' to ' + numFiles + ' - Check the server log')
                  resolve();  // reject() causes the pushPromise function to fail with `undefined`
              }  
          });      
      });
  }
  
  // Prepare promises array of ajax calls.
  function pushPromise(data, count, numFiles, totalRecords, moreFiles) {
      promises.push(promises.pop().then(function(){
      return postDbData(data, count, numFiles, totalRecords, moreFiles)}));
  }
  

  Changed the preSubmit to use the above promises solution and to eliminate the Datatables create from sending any data.

          .on( 'preSubmit', function (e, data, action) {
  
              if (action === 'create') {
                  var maxRows = 300;  // Max to submit at a time
                  var createData = data.data;
                  var rows = Object.keys(createData);
                  var totalRows = rows.length;
  
                  // Block UI while batch processing
                  console.log("Batch uploading files... Please wait");
                  addMsgDiv();
                  $("#msg").html("Batch uploading files... Please wait");
                  $.blockUI({ message: $('#msg') });
  
                  var count = 1;
                  var numFiles = 0;
  
                  // Batch process a group of rows
                  while (rows.length > 0) {
                      console.log('loop')
                      var batch = {};
                      var batchKeys = rows.splice(0, maxRows);
                      numFiles += batchKeys.length;
                      
                      var moreFiles = true;
                      if (batchKeys.length < maxRows) {
                          moreFiles = false;
                      }
  
                      // Move the rows to be processed into new object
                      // TO DO: determine if there is a more efficient way to do this
                      for (i=0; i<batchKeys.length; i++) {
                          key = batchKeys[i];
                          batch[key] = createData[key]
                          delete createData[key];
                      }
                      
                      pushPromise(batch, count, numFiles, totalRows, moreFiles);
                      
                      count = numFiles + 1;
  
                  }
  
                  // Complete with batches
                  // No data left but return true to close modal.
                  return true;
  
              }
          } );
  

  Using this solution allows for using the code and modals provided in the CSV Upload example but intercepts the Editor AJAX call to batch upload via promises.

  Kevin
• allan Posts: 61,723Questions: 1Answers: 10,108 Site admin

  February 2020 Answer ✓

  Hi Kevin,

  That looks like a nice solution - batch uploading in sequence with promises is probably about as good an option as it is going to get there at the moment.

  The only other option I can think of just now is to use preSubmit to trigger another Editor instance that will run is sequence, taking the next lot off a queue whenever submitComplete triggers. But that's effectively what you are doing with your promises anyway, just not using a second Editor instance.

  Allan
• kthorngren Posts: 20,309Questions: 26Answers: 4,769

  February 2020

  Thats a good idea. I'll have to try it out. Thanks!

  Kevin