Performance with 267 columns 117 rows

Performance with 267 columns 117 rows

jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
edited April 2014 in DataTables 1.9
I'm currently investigating a browser-freezing problem when using a large number of columns. I'm having a bit of trouble where to look because I'm rather new to browser/JS profiling.

However, I *think* what's happening is that the browser chokes on reflow. Because many of the bottlenecks are single statements - they are getting or setting sizes. Which I presume cause a storm of recomputations for something as complex as a table with so much columns. Maybe there's also a possible influence from the Scroller add-on (release 1.2.0) ?

For a smaller number of columns there seems to be no problem.

Is this a problem already familiar to somebody in here ?

Or any hint what alternatives I can try ? Preferably by maintaining that many columns. :-)

Thanks !

Jan

Replies

  • allanallan Posts: 61,451Questions: 1Answers: 10,055 Site admin
    Can you link to the page please so we can take a look. If you remove the computations and just put in dummy data, does that help?

    Allan
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    Unfortunately I am not at liberty to make it public. But I'll try to make something in Fiddle as a proof of concept.

    It would really be convenient if Scroller would also work horizontally. Because that's where the problem lies: too much markup.
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    I've prepared an example on JSFiddle: http://jsfiddle.net/DuA8v/4/

    First press "Add Big table" and then "Fill table". Wait for a few seconds to see the table appear. The spent time is displayed too. The "render" time is basically the duration of the DT call. The "build" time is the time spent to create the whole thing - apart from calling DT.

    The styling of the result is not really what we have here - but the intended result is there: Showing it takes 4/5s to render it. I'm no JS/CSS/... guru; hence my call for help to find out what's going on.

    To "simulate" the server, the DT settings object is converted into a string and then back into a JSON object. And only then fed to DT.

    Is there something obvious left to do to make this process go faster ?
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    edited April 2014
    Gave the wrong url. The correct is: http://jsfiddle.net/jan_goyvaerts/DuA8v/25/

    Sorry about that.
  • allanallan Posts: 61,451Questions: 1Answers: 10,055 Site admin
    > bPaginate: false,

    You are going to get no benefit from deferred rendering since you need all rows up front. So the biggest change I would suggest is enabling paging, or using Scroller.

    Makes your table draw in 600mS for me: http://jsfiddle.net/DuA8v/26/

    Allan
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    edited April 2014
    I've updated the jsfiddle sample - it is using Scroller and is little more parameterizable.

    http://jsfiddle.net/jan_goyvaerts/DuA8v/28/

    The total amount of rows is extended to 5,000. As is our use case. Reducing the columns does make it much more responsive.

    Maybe this is shedding some light ? I guess I'm making a stupid mistake somewhere... :-/
  • allanallan Posts: 61,451Questions: 1Answers: 10,055 Site admin
    No stupid mistake, it just that it is taxing both DataTables and the browser!

    Using DataTables 1.10 gives a performance boost of about 1.5 seconds. Disabling autoWidth gives about another 1.5 seconds.

    Doing a profile on the table creation shows:

    - 32% of the time is getting the scroll top position
    - 30% of the time is getting the offsetWidth for the scrolling elements
    - 17.5% Getting the widest element for sizing (auto width)
    - 9% For initialisation

    Any suggestions for how to improve any of them are very welcome. At the moment I would suggest using server-side processing. DataTables can't virtualise the columns in the same way as it can for the rows with Scroller, so the less data you ask it to process the better.

    Allan
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    Doing it with a web service was my next attempt. To check the difference. I'll include the 1.10 and see if autowidth can be disabled for our usages.

    I think it might be a solution to actually disable Scroller on a case basis. When the table is too wide, then let's use the classic paging mechanism. With the navigation buttons and one page in memory only. Even if there are thousands of rows.
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    Thinking about it: A violent hack that *seems* to be working for me is to run _fnScrollDraw just once. Because the result of some measures was always the same.

    [code]
    function _fnScrollDraw(o) {
    var firstPass = $(o.nTable).data("opt-first-pass") === undefined;
    if (firstPass) {
    console.log("Running _fnScrollDraw()");
    var start = window.performance.now();
    $(o.nTable).data("opt-first-pass", false);
    _fnScrollDraw_original(o);
    var duration = window.performance.now() - start;
    console.log("Running _fnScrollDraw() took " + duration + " ms");
    }
    else {
    console.log("Skipping _fnScrollDraw()");
    }
    }
    [/code]

    But I'm not suggesting it's useless to do so ! :-) But maybe it means there is an opportunity to win time somewhere ?
  • allanallan Posts: 61,451Questions: 1Answers: 10,055 Site admin
    Agreed - i'm certain that there optimisations that can go in there.

    Allan
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    Anothing thing I was wondering about is reusing existing the existing instead of removing and adding them each time all over again. Wouldn't that cut in the cost of rendering ?
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    edited April 2014
    Maybe it's faster also if Scroller only adds the rows to the DOM when they're about to become visible ? Instead of adding the whole prefetch buffer at once ?

    Something I hacked - but failed to do it properly - is to put the rendering in a timeout function:

    (1) Put the rows into a buffer array.
    (2) Schedule a timeout function:
    (2.1) Remove the 10 first rows from the buffer
    (2.2) Add the removed rows to the DOM
    (2.3) If the buffer isn't empty => GOTO (2)

    It's actually cool - the browser responds much faster and the rows appear while you at it. But something went wrong somewhere. :-)

    I'd like very much to have a try but I'm currently a bit overwhelmed at work.
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    As an example of asynchronous processing. It works perfectly when letting it run to its end. Otherwise it's very easy to mess it all up.

    Also the effect is lost when Scroller only resets the viewpoint AFTER everything has been drawn. :-)

    So, there's a little tinkering left here...

    [code]
    function _fnDraw() {
    ......
    ......

    var body = $(oSettings.nTBody);
    body.children().detach();

    // HACK: Draws the rows individually in background so *seems* less slow. It replaces the commented code underneath

    asyncDraw(anRows, oSettings, body);

    // body.append( $(anRows) );
    // /* Call all required callback functions for the end of a draw */
    // _fnCallbackFire( oSettings, 'aoDrawCallback', 'draw', [oSettings] );
    // /* Draw is complete, sorting and filtering must be as well */
    // oSettings.bSorted = false;
    // oSettings.bFiltered = false;
    // oSettings.bDrawing = false;

    }

    /**
    * Hack to draw the rows asynchronously. It'll take at least just as much time,
    * but the first response comes a lot faster.
    *
    * @param anRows The rows left to add
    * @param oSettings Settings
    * @param body Element to append the rows to
    */
    function asyncDraw(anRows, oSettings, body) {
    if (anRows.length == 0) {
    _fnCallbackFire( oSettings, 'aoDrawCallback', 'draw', [oSettings] );
    oSettings.bSorted = false;
    oSettings.bFiltered = false;
    oSettings.bDrawing = false;
    }
    else {
    window.setTimeout(function() {
    var page = anRows.splice(0,1);
    body.append($(page));
    asyncDraw(anRows, oSettings, body);
    },0);
    }
    }

    [/code]
  • allanallan Posts: 61,451Questions: 1Answers: 10,055 Site admin
    Looks like a good idea. There will likely be some quirks due to the async nature of the code, but it might be worth exploring further. Patches to Scroller are very welcome :-)

    Allan
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    Is there an address I can send the modified .js files to ? I've made a number of other changes but they're not 100% just yet.

    I guess won't divulge company secrets by sending them to you. :-)
  • allanallan Posts: 61,451Questions: 1Answers: 10,055 Site admin
    A pull request on github would be great: https://github.com/DataTables/Scroller . Otherwise allan @ this domain will get to me.

    Allan
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    Was what I sent any useful ?
  • allanallan Posts: 61,451Questions: 1Answers: 10,055 Site admin
    I don't think I received anything I'm afraid.

    Allan
  • jan_goyvaertsjan_goyvaerts Posts: 31Questions: 0Answers: 0
    Just sent it again. This time the JS source code is in a zip file.
This discussion has been closed.