Add an option to scroller to leave the already loaded rows

WEHA

I know about the pipelining, but this does not seem to work well with scroller (I don't want pages itself)

I also know you're going to say: for n rows you don't need scroller.
To that I say: yes the browser can handle it, that does not mean the server can load them as fast from the database nor does it fix the extra delay you have because the response is heavier.

I just want everything to be snappy and this would solve 2 problems for me:
- faster scrolling when going back
- no shifting while scrolling when some rows have a different count in lines
- maybe less problems selecting multiple rows over extra draws

Thanks

allan

Scroller should work with pipelining. If it isn't for you, please link to a page showing the issue.

Thanks,
Allan

WEHA

Ok, I want to create one but neither live.datatables & jsfiddle allow me to use an ajax source.
I tried https://datatables.net/examples/server_side/scripts/server_processing.php
Do you have a server processing source I could use for live.datatables?

colin

You could use this as a starting point - it's using serverSide,

Colin

WEHA

What I see is it all loads fine until you scroll back up, it needs to load again for some reason.
Which is just what I want to prevent.
The first time it didn't show this problem in the example, it's probably more apparent when there are like 500 rows or something.
Maybe you can already see the culprit?

http://live.datatables.net/havobiru/2/edit

colin

Thanks for that test case. That's a known issue that's been resolved in the nightly builds - see your example updated here to use the nightly build. You can use that nightly build until the release is made.

We aim to release this in the next couple of weeks (we'll be releasing this and many other extensions).

Colin

WEHA

Well I notice the same behavior in the nightly build.
At some point when I scroll to the top there are no rows visible at all.

Also, isn't that background for when there are no rows?

I see server request scrolling back up, or does this have to do with pages: 5?
I want loaded rows to remain but I don't want to initially request 5 pages.
What should this setting be in that case?

Maybe my expectation is too high?
What I expect but is still happening:
* Scrolling down will not have scrolling offset changes because of single & double rows as it's loaded and can be accounted for (not in test data so unsure)
* Scrolling up will show the already loaded rows
* * no requesting the server for more data as it's already in memory
* * no "loading" type events like rows coming in to view with a delay.

colin

At some point when I scroll to the top there are no rows visible at all.

I've tried reproducing that with the same test case from before, but I'm not having any joy, I'm not seeing that behaviour. Please could you give steps on how to reproduce.

Maybe my expectation is too high?

Possibly. As Scroller says on its front page, it offers a scrolling container - the size of which is determined by scroller.displayBuffer. If the scrolling goes outside of the rows within that container, whether you're scrolling up or down, then a request will be made to the server for more. No caching is performed on the rows previously seen.

Colin

WEHA

Just do some scrolls down and up, all the way down and then all the way up.
First it tried to reproduce it happened on the second scroll to the top.
Second time I had to scroll up and down some more but eventually it will get to that stage.

So if I understand correctly what I want accomplish is currently not possible?
I guess that means this will stay a feature request
What I'm asking is not to cache it in the background but to leave the rows in the container. It's probably not a hard thing to do?

colin

I've scrolled up and down about a dozen times without seeing the issue - I also did the same last time at around the table, all without being able to reproduce. I'm using Chrome Version 84.0.4147.105 (Official Build) (64-bit) on Fedora 32. Can you give some information about your environment, please.

I'll let Allan comment on the feature request,

Colin

allan

You are correct - it isn't currently possible.

It's probably not a hard thing to do?

Unfortunately it is for two reasons:

1. We use a single table to display the rows, so there can't be any disconnect between sections - it would end up just being one large table. The only way to resolve that would be to use div elements for each row, similar to what SlickGrid does, and that is a semantic change I don't want to make in DataTables.
2. Scroller uses non-linear scrolling to allow for massive data sets. While placing elements in position is easy for a linear scroll with small data sets, we have to use non-linear scrolling due to the browser not being able to have infinite height containers. This means that while scrolling we'd need to reposition cached elements.

So it is possible to do, but not something we are going to be implementing I'm afraid.

That said:

At some point when I scroll to the top there are no rows visible at all.

That should not be happening. I've also not been able to reproduce that issue I'm afraid. When I scroll to the top of the table it is always showing the rows at the table start for me - Firefox 79.

Allan

WEHA

Thank you for explaining, although I don't understand everything you are saying.
1. Do you mean to say with "disconnect" that you can scroll a few pages and that scroller then does not retrieve all rows between the last visible row and the new calculated end row? Is it a problem to just load all rows from that last limit to the new limit? Which wouldn't be a problem, for me at least.
2. Well it wouldn't be for large datasets so this would not be an issue. If it's documented that this feature only supports up to x single rows or something? What is the height limit?

I'm also using Firefox 79, on Windows 10.
I just tried it in safe mode and the same issue occurs.
Updated to Firefox 80 & in safe mode, same issue.
Chrome does not have the issue apparently.

colin

As Allan said, it stores continuous rows, up to the size of scoller.displayBuffer. The extension is open source, so you're welcome to modify it to suit your needs.

I just tried http://live.datatables.net/havobiru/2/edit on Windows 10 with FF 79, and I'm still not seeing the issue. I've been scrolling around the table for over a minute. When you say you tried it in safe mode, is that a private window? If not, could you try that please, in case any extensions are affecting it,

Colin

WEHA

Yes but displaybuffer also defines how much rows are requested from the server.
I'm terrible at javascript so it will take me ages to even understand what is going on in the code, let alone change it.
But ok, if my thinking is not correct and it remains a rewrite then it's not feasible.

Safe mode means no extensions, themes & hardware acceleration.
Private is just normal with maybe fewer enabled extensions.
Did you throw in a few scrolls to the complete bottom and scroll back up in one go?

colin

Yep, all the way down, all the way up several times. Are you using a mouse? And releasing? Or trackpad? Any more information you can give would be helpful, as we're at a loss here.

Colin

WEHA

Yes with a mouse. From what I can tell it's not needed to go all the way down.
It seems to happen randomly when I get to the top again.
Not sure what you mean with releasing? Unless you mean on a trackpad.

Tried on Windows 10 2004 & 1909
Just tested on Windows 2012 R2, same issue.

It seems that the top rows aren't even in the html at that point:

This is another situation where the next row is visible and I think also in the location it should be:

Let me know if there's anything else I can give you. I'm not sure at this point what else I can say

Fyi: I also don't receive e-mail notifications of replies, even though it's enabled in forum settings. It does appear in my forum notifications inbox.

allan

Are you able to send us a screencast video of the issue? We aren't seeing it here (an empty space, after you stop scrolling).

I also don't receive e-mail notifications of replies, even though it's enabled in forum settings

Might be going into your spam folder?

Allan

WEHA

https://ttprivatenew.s3.amazonaws.com/pulse/weha/attachments/14569389/TinyTake03-09-2020-04-47-00.mp4

It happens at 24 seconds.
Recorded with Firefox in safe mode

No, notifications are not in my spam folder, nor does my server receive any attempt to deliver a notification.

allan

Thanks! Sandy has just been scrolling for a few minutes, but hasn't been able to reproduce the issue unfortunately.

Going back for a moment - are you able to reproduce that issue without pipelining?

With Scroller, rather than using pipelining, it would be more efficient to just increase the buffer size.

Allan

allan

Also, how many rows of data are you planning on showing in the table? Is server-side processing required?

Allan

WEHA

That's just weird, I have been able to reproduce it on 3 different systems.
Yes it also happens without pipelining: http://live.datatables.net/havobiru/5/edit

Increase buffer: Well for some tables that would be possible but I have lists that require significantly more processing to produce so I don't really want to increase the scroller buffer size.
And it's more to increase the experience for rows that already have been loaded.
For example:
- selecting rows beyond the buffer size
- faster scrolling back
- no repositioning because of single and double rows (as in different row heights)

Server processing is required, loading the whole set takes too long and is usually not necessary as well.
We're talking about maybe 10000 to 15000 rows. Most of them are in the lower thousands.

allan

for some tables that would be possible but I have lists that require significantly more processing to produce so I don't really want to increase the scroller buffer size.

I don't quite get this point I'm afraid. What is the difference between Scroller requesting 100 rows and the piplining requesting 100 rows? Unless you mean that the drawing of the table rows is an expensive operation?

Allan

WEHA

I didn't say there was? It's also not what pipelining is for, unless I'm mistaking?
Increasing the buffer does not solve my problem and pipelining does not do what I want to do from what I can tell just something in between.

So again, in short, I would like to load the rows gradually but let them remain in the browser so the experience much more fluent.
You said it wasn't possible to store this in the table element for some reason.
A reason which I don't comprehend but I don't know the inner workings of datatables.

The way I see it is you just leave the loaded rows and load further rows into it.
If the user scrolls much further it should load rows starting from the last loaded row and not just the current view of rows to remain consistent.
But that's probably oversimplifying.

I understand this could be an edge case for you or simply too much work.
No worries, it is what it is

allan

The main issue is that we can only draw consecutive rows. If you could position each row independently, then you could do as you suggest, and that is exactly what SlickGrid does for its virtual scrolling by using div elements. We use a div though.

The closest we could get would be to cache the data - which pipelining could be modified to do if you were so inclined :-).

Allan

WEHA

I only want consecutive rows so that's why I don't understand what the issue is

I'm still not receiving notifications