DataTables

I've just implemented this in a fork of the 1.8 development version of DataTables, which you can get from here: http://datatables.net/dev/jquery.dataTables.delayedRendering.js . I don't have access to an IE machine at the moment, so can't test the impact on that - but there is almost no impact on Safari with 5000 rows... Would be interesting to see what you find.

Regards,
Allan

Hi Jake,

Thanks very much for the donation :-). Very much appreciated!

How much of a performance improvement are you seeing? When you say a JSON result set of 25 records in 3 seconds - that's rather slow I would have said...

Allan

I suspect what is happening is that fnSetColumnVis is trying to find the TD element which hasn't yet been created. I think I'll need to get back to you on this one - it looks like that might be a little tricky (or at least require a little thought :-) ).

Allan

Hi Allan,

Typo on my part- I meant to say 25k records!

It's EXTREMELY FAST now with IE8 and blazing fast with IE9. This is an incredible improvement!

Regards,
Jake

@Ben: Ah - it's ColReorder that is triggering this! Okay that narrows it down - thanks :-)

@Jake: Cool! I'm happy with that!!!

@Scott: All you need to do is grab the forked code from the above link and then use either a JS array or Ajax source for your data source: http://datatables.net/examples/data_sources/ajax.html or http://datatables.net/examples/data_sources/js_array.html . The way it is implemented is that DataTables will only create the TR/TD nodes for the rows which get displayed (a bit like server-side processing). Unlikely server-side processing however, once a node is created, it is retained, so it doesn't need to be recreated if shown again.

The upshot (and what makes me ever so slightly nervous - so I need to think on this a bit) is that not all nodes are available at all times. So functions like fnGetNodes() will return only nodes which have been pre-rendered. If you use fnGetNodes to attach events it simply won't work - live events should be used instead. It's not a problem if you are aware of this, there is nothing that can't be dealt with as a result, I'm just wondering if it might be confusing.

If I implement this in DataTables 1.8 (which if it is coping with 25k rows, then it's very very tempting) then it will almost certainly be with an optional flag to enable this feature. bDeferRender perhaps...

Regards,
Allan

I am going to try to implement this.

One question though, is the delayedrendering automatic? Or is a method we have to call on datatables. In looking at the source code, it appears to be automatic and default. Is this correct?

Thanks,

Scott

@Scott: In the development version linked above it is fully automatic. When I integrate it into the trunk, I will probably make it optional using a parameter such as bDeferRender. Would be interested to know how you get on with it.

@Jake: I'll wrap up what I'm currently working on in the DataTables core and then implement this properly into the trunk and make a decision then. If it passes all unit tests and doesn't make anything too complicated, then it will go in. It sounds like it really does give a significant improvement - excellent :-)

@mg: This change doesn't effect the speed of sorting at all. It effects the initialisation time of the table. WHere before it used to create the DOM for all rows and columns during initialisation, it would now create the elements only when needed. The data used for sorting and filtering is still in its internal cache and those algorithms are unchanged.

Allan

@Scott: How many rows are you using? As I mentioned in my last post (directed at mg) this change doesn't effect sorting speed. DataTables does indeed have it's own internal cache for sorting - but there is a limit to how many statements IE will process before throwing up it's slow script warning - 5'000'000 ( http://support.microsoft.com/kb/175500 ). Sounds like a lot, but there is a lot going on with the sorting. It's possible it can be optimised some more, although I'm not sure where at the moment. DataTables 1.7.5 introduced a number of optimisations, so I'd make sure you are using the latest version first.

Allan

I've now built this into DataTables core, given the benefit in speed that it gives - which is fantastic :-). This delayed rendering of the TR/TD elements can be enabled by setting bDeferRender: true at initialisation time. The reason it isn't on by default is that it changes the old behaviour of DataTables, and I want to retain backwards compatibility with v1.8. Following on from the v1.8 release, I will consider making this on by default (possibly without the option of disabling it) if it proves successful for v2 onwards. I've also set up unit tests for this new option and it passes all tests for all data sources (2500+ tests now :-) ).

For maximum speed you will want to disable the sorting classes which highlight the currently sorting column (bSortClasses). I'll write an article about the performance / features tradeoff for DataTables soon.

I'd be most interested to know how you all get on with the new changes! You can pick up the nightly version with this change from the download page: http://datatables.net/download/ .

Regards,
Allan

Hi Scott,

Just a sanity check - you've got "bDeferRender": true in your code now do you with this new version? I don't think you are "doing something wrong obviously" - 29000 records is a lot of data to process, and it takes time to do so. IE allows 5 million instructions before a timeout is issued, which is 170 odd instructions per record for your instruction set - it might sound like a lot, but function calls etc all stack up quite quickly.

I have been thinking of a way to speed up sorting quite significantly (even in IE), but I think it will need to wait until 2.0 since it will not be backwards compatible with the current sorting plug-ins. Something to watch out for in future :-). For now though, have you got bDeferRender: true and bSortClasses: false?

And yes, there would be no difference to DataTables between getting a JS source or an Ajax source.

Allan

Hi Scott,

Thanks for the heads up about the two issues.

TableTools: It makes reference to aoColumns[i].nTf - which is now incorrect as nTf has been removed from the columns object. So TableTools will need an update for the 1.8 DataTables release. The work around for the moment is to set bFooter:false for the button configuration. I think my "fix" will be to strip out the footer from the TableTools output.

Sort classes and deferred rendering: Good find. Thanks for that. I'll fix that shortly.

Thanks for the testing!

Allan

Sure no problem - at some point I'll write an article about the various data source options that DataTables offers - before for now:

By default when DataTables loads the Ajax data source, it will process each row for inclusion in the table. This processing involves creating the TR and child TD nodes for the row, as well as rendering the content (if fnRender is given), type detection and various other things. When done the row is then read for sorting, filter or whatever else DataTables needs to do - most of which does not require the DOM elements (since that's very slow).

What bDeferRender does is to delay the building of the TR / TD elements until they are actually needed (i.e. for display). So, for example, if you are displaying 1000 rows, then it will need to create the elements for 1000 rows. If you are displaying only 10, then it will need to create the elements for only 10 rows - regardless of your data source size. However - the data is still needed for sorting, filtering etc, and these processes are unaffected. When DataTables draws a row on the page, it will check if it has been rendered or not already - if not then it will render it there and then.

Thus, the idea that Jake originally put forward saves a big chunk of time in the creation of the DOM elements, since it is spread out over time, rather than all done at the same time. It will not, however, magically make the Javascript processor in IE8- in any way considered to be "fast" by modern browser standards. The IE6 engine in particular is terribly slow. 29'000 is a lot of data to process, and if your Javascript engine is slow then there is no amount of optimisation that will make it decent. There are things I can do to help it, and things I plan to do, but it is a decade old browser now :-)

Regards,
Allan

Hi, I am running 1.8.2 and I see this issue on IE7 which is standard in our company. I have about 1500 rows in the table and it is paginated at 125 per page and it is very slow to unuseable. I do the same thing in Firefox and it renders fast(in a second or so).

I am not a developer but figured out how to use datatables to display my data and I love it.

Here is the config I am using
<script type="text/javascript" charset="utf-8">
$(document).ready(function() {
$('#example').dataTable( {
"bPaginate": true,
"sPaginationType": "full_numbers",
"iDisplayLength": 125,
"bLengthChange": false,
"bFilter": true,
"bSort": true,
"bDeferRender": true,
"bInfo": false,
"bAutoWidth": false } );
} );
</script>
Any updates on how to fix this would be appreciated.

Thanks,
Mike

Hi Allan,
[deep bow for creating this tool]. Since I embed data in the <tr> tag that I use for later Jquery processing, I am returning the whole <table> from the server. Since the bDeferRender only helps if I am using raw, ajax sourced data, I am out of luck, is that correct?
Best
Mike

Thanks for the prompt response. I am putting the whole table on the page :(
My stats are 16 seconds for 290 records.

I could rewrite my code to send this hidden data back as invisible columns, but embedding the hidden data in the <tr> element was rather convenient (i.e., I don't have to search all the <td>s for hidden data). That rewrite would be rather painful, though

If you are able to compensate for the inadequacies if IE, I am committing to a $50 donation

Details below
I get the <table> with this call

 $.get("run.php", QueryString, function(data)
{
  [...]
  $('table.dataTable').InitDataTables(TableName, null, oQueryParams);
  [...]
}
$.fn.InitDataTables = function( TableName, SelectAry, oQueryParams ) {
[...]
  oGlobalData.oTable = $("#"+TableName).dataTable( {
         "aLengthMenu": [[25, 50, 75, -1], [25, 50, 75, "All"]],
         "iDisplayLength": 25,
         "bJQueryUI": true,
         "bAutoWidth": false,
         "bDeferRender": true,
         "sDom": '<"H"fp<"TableName"><"BTNS">>t<"F"li>'
    });          
[...]
}

Hi Allan,

With DataTables disabled, it comes in about 1 second. I have the display set to 'block'. Just as an experiment, I set it to none and then returned it to block after initialising the DataTable and that took the time up to [drum roll please] 40 seconds! Sorry, I don't have a public link up yet.
Since I am refactoring to replace PHP with python/django, I think I'll change the logic to send the extra info in <td>, so I can take advantage of the bDeferRender option. Many thanks for answering (I dropped a few greenbacks in the tip jar yesterday)

Best
Mike

Err, I thought I was linking to 1.8.2 but was actually using 1.7.5. With 1.8.2, 295 rows come back in 3 seconds, not 16 seconds.

Sorry for the misinformation...