Search  Rules  Login
TheBat English » The Bat! - Configuring the E-mail Client
Pages: 1 2 Next
RSS
Windows 22/23h2 sloooows down after is TB! 11.x started and running untile next reboot
 
#1
0  
Am I the only person who experiences this? I can not find any solution to this, reinstalling TB! 11 helped just for a moment and then Windows becomes laggy again.  
 
 
 
#2
0  
What does your Task Manager say? Mine indicates a 4-5MB/s load on the disk when idling, which is more that any other program:

https://imgur.com/a/n6Fu6nb

I imagine this could choke the system if it's on a regular hard drive also running other stuff.

EDIT: it just went up to more than 7MB/s, and I'm not doing anything.
Edited: Miloš Radovanović - 31 December 2023 18:25:00
 
 
 
#3
0  
https://ibb.co/wCNDyLn

But I do not think it is the problem. After shutting down TB! Windows still does not work correctly - needs a reboot. Trying to figure it out with downgrading TB! - let's see if anything changes.

After downgrade:
https://ibb.co/8gqHSxB

Everything works just fine for now, no lags - all apps running just after click.
Edited: Marek K. ZBOROWSKI - 31 December 2023 19:28:49
 
 
 
#4
0  
Quote
Miloš Radovanović wrote:
What does your Task Manager say? Mine indicates a 4-5MB/s load on the disk when idling, which is more that any other program:

https://imgur.com/a/n6Fu6nb

I imagine this could choke the system if it's on a regular hard drive also running other stuff.

EDIT: it just went up to more than 7MB/s, and I'm not doing anything.
How do you do that? I have 6 entries all about 20 mb

https://imgur.com/a/JBqU4e7
 
 
 
#5
0  
Quote
Rick Gijsbers wrote:
How do you do that? I have 6 entries all about 20 mb

Me? Or Ritlabs?

Any clues what to do?
 
 
 
#6
139.3368  
I experienced slowdown as well on my computer. In my case, TheBat and the other applications are on HDD. After every restart, for some reason TheBat starts reindexing the entire mail collection and, I believe, it is rebuilding index for the new feature "conversation view". On my computer, TheBat! thrashes my HDD for about an hour after each restart with close to 50% disk utilization that cripples the rest applications when they want to read/write something to HDD. I can understand that it can be one-time trouble to build the initial index after the upgrade, but rebuilding indices after every restart from scratch - that I do not understand.
There was some advice to turn off new "Conversation panel" in the viewer setting, but it did not help. I had to rollback to 10.5.3.2 and now I am waiting for the fix (I hope it will be fixed).

Here is the picture of TheBat thrashing my HDD - at the blue background a file manager shows TheBat files sorted by modification time, so you can see it constantly write to index.db and index.db-journal in turns. The disk utilization is 49%: https://imgur.com/a/kdtvBIY
Edited: Vladimir Markelov - 02 January 2024 21:34:44 (Added picture)
 
 
 
#7
7.3921  
Quote
Rick Gijsbers wrote:
How do you do that? I have 6 entries all about 20 mb https://imgur.com/a/JBqU4e7
That is RAM usage, those are fine. What is troublesome is disk usage which in your case is 0 MB/s, which is also fine.
 
 
 
#8
0  
Quote
Vladimir Markelov wrote:
I experienced slowdown as well on my computer. In my case, TheBat and the other applications are on HDD. After every restart, for some reason TheBat starts reindexing the entire mail collection and, I believe, it is rebuilding index for the new feature "conversation view". On my computer, TheBat! thrashes my HDD for about an hour after each restart with close to 50% disk utilization that cripples the rest applications when they want to read/write something to HDD. I can understand that it can be one-time trouble to build the initial index after the upgrade, but rebuilding indices after every restart from scratch - that I do not understand.There was some advice to turn off new "Conversation panel" in the viewer setting, but it did not help. I had to rollback to 10.5.3.2 and now I am waiting for the fix (I hope it will be fixed).Here is the picture of TheBat thrashing my HDD - at the blue background a file manager shows TheBat files sorted by modification time, so you can see it constantly write to index.db and index.db-journal in turns. The disk utilization is 49%:  https://imgur.com/a/kdtvBIY

Thanks for sharing, that's it! And there doesn't appear to be a way to turn this indexing off in v11.0.2. Mine lasts 10-15 min after every TB startup, but it's not as noticeable since I have an SSD drive. However, with my usage patterns this is bound to shorten the lifespan of my drive, as it does around 7GB of disk writes each time I start TB. So I also rolled back to v10.

Ironically, this extra indexing seems unnecessary as TB already has everything that's needed. The "conversation view" could be made to simply scroll through messages as they are in the message list. And the message list can be configured to show threads that include messages from multiple folders (with a message list tab or virtual folder). I set up such a message list tab and it does it's job pretty fast on my ~20GB message base (~4 sec. to load initially, and after that scrolling through the message list and viewing messages is instant).
 
 
 
#9
0  
I reported these issues to support, let's hope they make it right soon. I kinda like the concept of the conversation view, but in it's current form it just doesn't cut it for many users including myself.
 
 
 
#10
0  
I have the same problem and I have contacted Rilabs, the response has been that they have no evidence of it happening to anyone else.
 
 
 
#11
0  
Quote
Leopoldo Garcia wrote:
I have the same problem and I have contacted Rilabs, the response has been that they have no evidence of it happening to anyone else.

I wrote to them on January 3, and on January 6 I got the reply: "Thank you for your message and feedback, we will forward your suggestion to the developers."

And this is what I wrote:

"There appear to be serious problems with indexing for the new conversation view: from excessive disk writes in my case (10-15 minutes after each TB startup, writing ~7GB to my SSD which will shorten it's life span considering my usage patters) to lockups and extreme slow-downs for users with HDDs. Also, it seems the indexing is performed from scratch after each startup, and there is no way to turn it off (it is performed even when the classic preview pane is used). Please see here:

https://www.ritlabs.com/en/auth-forums/forum4/topic16259/

Permit me to offer a suggestion. From what I can see the index for the conversation view is created with SQLite, which appears to be inferior for this purpose when message bases are large (mine is around 20GB). On the other hand, existing TB indexes for individual folders are extremely fast and reliable, and if I understand things correctly can be efficiently combined when making message list tabs and virtual folders that collect messages from multiple folders. For example, I created a message list tab called "Conversations" that applies a threaded view mode to a selection of almost all my message folders, and after the initial click on this tab it takes ~4 seconds for the messages to show up, after which all accesses to the messages in the tab are instant. I presume those 4 seconds are the time needed for the folder indexes to be combined, which is an order of magnitude faster than the implementation using SQLite. I urge you to consider using the tried and tested indexing technology you already have.

Forgive me for being so bold to take my suggestions one step further. Please consider making the conversation view into a simple scrollable view of whatever messages are in the message list (at least as an option). That way the new viewer can still present conversations using a tab like the one I described above, or view messages in any other way a user sees fit. It would add versatility to the viewer, as well as make the implementation simpler (at least from what I can see as a user with a computer science background).

Thank you for your time."
 
 
 
#12
0  
I haven't been able to get SysInternals' System Monitor (SysMon) to work on my current PC yet, but that would be the perfect tool to figure out which files, specifically, The Bat is writing to all the time. It might give a deeper insight into what is going wrong.. maybe you guys could give it a try.
I volunteer as a moderator to help keep the forum tidy. I do not work for Ritlabs SRL.
 
 
 
#13
0  
I'm seeing a variety of other issues for which I've opened tickets for, but even with 11.0.3 I don't see this indexing issue.  FWIW, I'm running 11 in an isolated VM with nothing else running aside from the OS.  In addition, the two accounts I'm testing are IMAP and the largest only has a little over 3,000 emails.  I can see indexing activity on every start by watching the data folder, but my index file is only 3mb.  It seems to be 'updating' it rather than rebuilding since it doesn't start off at 0 bytes on each start.  
 
 
 
#14
0  
Quote
Daniel van Rooijen wrote:
I haven't been able to get SysInternals' System Monitor ( SysMon ) to work on my current PC yet, but that would be the perfect tool to figure out which files, specifically, The Bat is writing to all the time. It might give a deeper insight into what is going wrong.. maybe you guys could give it a try.

Thanks for the tip Daniel. Even without special tools I was able to figure out that TB was doing ~7GB of disk writes on each startup, and Vladimir showed above which files were being written. I was left with a large file called index.db in my TB data folder, which after inspecting in a simple text viewer revealed that SQLite was being used to create a relational database and index for messages. All messages from all accounts and folders, it seems. Still boggles my mind why the devs would introduce something like that when they had everything that was needed at their fingertips. It took me a couple of clicks to create a "Conversations" message list tab which does exactly the same thing (in terms of index creation), and takes 4 seconds to execute, all in RAM.

I guess they may have "bigger plans" for this conversation view and want to dich some "limitations" of the current indexing implementation, but it wasn't a great start, to put it mildly.
Edited: Miloš Radovanović - 07 January 2024 20:30:11
 
 
 
#15
0  
Quote
fletch wrote:
I'm seeing a variety of other issues for which I've opened tickets for, but even with 11.0.3 I don't see this indexing issue.  FWIW, I'm running 11 in an isolated VM with nothing else running aside from the OS.  In addition, the two accounts I'm testing are IMAP and the largest only has a little over 3,000 emails.  I can see indexing activity on every start by watching the data folder, but my index file is only 3mb.  It seems to be 'updating' it rather than rebuilding since it doesn't start off at 0 bytes on each start.

I use POP3 and download all my messages, but that is probably irrelevant. Across two accounts I have ~140,000 messages taking ~20GB of space. I was left with a 200MB index.db file after rolling back to v10, but I don't know whether that's a complete version or one I cut off by shutting down TB before the indexing was finished (it would take 10-15 minutes).

To be clear, v11.0.2 was totally usable for me, I just don't want to have 7GB of useless disk writes (measured using the Task Manager) every time I start TB. With my usage patterns, I calculated that alone would do as many disk writes in 6 months as I had in the 2.5 years of using my current PC. That's a five-fold increase in the amount of disk writes, I don't want my SSD to die too soon.
Edited: Miloš Radovanović - 07 January 2024 20:50:34
 
 
 
#16
142.5982  
I just imported my remaining IMAP accounts, so only about 20,000 records.  Makes you wonder what kind of different environments they develop and more importantly test in.  My index grew to 10mb.  I guess it builds/updates so quickly for me that I don't see the thrashing.  
 
 
 
#17
0  
Quote
fletch wrote:
I just imported my remaining IMAP accounts, so only about 20,000 records.  Makes you wonder what kind of different environments they develop and more importantly test in.  My index grew to 10mb.  I guess it builds/updates so quickly for me that I don't see the thrashing.

Thanks for the update fletch, this is interesting. I may have been wrong on two fronts:

1. The difference between POP3 and IMAP could be relevant. I use POP3 for the account with over 130000 messages.
2. SQLite may not be inherently bad (slow), it could be a RAM allocation issue. If the message base (and consequently index size) is larger than some threshold, SQLite may be paging to disk instead of doing everything in RAM, causing the thrashing and slowdowns.

I never worked with SQLite, but a quick Google search revealed some interesting quotes:

"How large will your database get in the future? SQLite requires too much memory to run if the database is over 1GB in size (256 bytes of RAM for each MB of database space)."

"The default suggested cache size is -2000, which means the cache size is limited to 2048000 bytes of memory. The default suggested cache size can be altered using the SQLITE_DEFAULT_CACHE_SIZE compile-time options."

I have a lot of work to do in the next couple of days, but when I grab some time I'll do some tests to determine if there is a message base size threshold where the SQLite index starts behaving badly.
 
 
 
#18
0  
I'm currently evaluating v11 and am also seeing continuous disk writes to the index.db file. My setup consists mostly of POP3 accounts with several thousand messages each.

What's more: I'm using encrypted mail databases, yet the index.db contains tons of plaintext data!

So it looks like I'll be staying on a previous version until all of this is resolved.
 
 
 
#19
0  
Version 11.0.3.1 just came out, with "An option to disable the message base indexing (increases performance on slow systems, but the cross-folder conversation thread view becomes unavailable)". So, v11 should be usable again for folks with these problems, only with the classic preview pane of course.
 
 
 
#20
0  
Increases performance on slow systems - really - it didn't slow my system but was absolutely hammering the hard drive which even if performance isn't impacted is not acceptable, this 'index' whatever it is shouldn't need to be rebuilt in its entirety every time the system is booted.


I think also they haven't considered the size of some repositories, I have multiple emails handled by the bat - 20 or so accounts totalling 12.4GB.

I also don't delete anything until it is at least 8 years old (and sometimes not even then) so I store into an archive account annually - you wouldn't believe how many times that has been useful.


Would you say it is safe to try again though ?
 
 
 
#21
0  
Quote
Neil Kirkland wrote:
Increases performance on slow systems - really - it didn't slow my system but was absolutely hammering the hard drive which even if performance isn't impacted is not acceptable, this 'index' whatever it is shouldn't need to be rebuilt in its entirety every time the system is booted.
I also found that statement a bit disingenuous, but let's not dwell on that.

Quote
Neil Kirkland wrote:
Would you say it is safe to try again though ?
One can never be sure. I will try it during the week, but in my case v11 was usable from the start since I have an SSD.

Support informed me that the setting is called "Disable Indexer for performance on slow systems" in the "Options\ Preferences\ General" menu.
 
 
 
#22
0  
Quote
Miloš Radovanović wrote:
Support informed me that the setting is called "Disable Indexer for performance on slow systems" in the "Options\ Preferences\ General" menu.

Have they explained how long this indexing should ordinarily run?

Does it continue to run indefinitely even when no new mail comes in and you do not move, copy or delete any messages?

It doesn't make any sense to me, and rather than offering an option to disable the whole feature, I think they should figure out why it continues to run even after its goal (whatever that may be) has been achieved.

Or, maybe the indexing is triggered every time that a small change has been made to the message base, but it is incredibly inefficient. Or, it doesn't update its indexes when a change is found, but it rebuilds them completely each time. Either way, Ritlabs seems to assume that this bug is a feature, and they're most likely wrong about that.

It might be helpful if they offer you an alpha version that enables tracing (maybe through an optional command line option - it would make the program even more unworkably slow), so each SQLite query that's run will also be recorded in a log file for bug hunting purposes.
I volunteer as a moderator to help keep the forum tidy. I do not work for Ritlabs SRL.
 
 
 
#23
0  
Quote
Miloš Radovanović wrote:
One can never be sure. I will try it during the week, but in my case v11 was usable from the start since I have an SSD.

Support informed me that the setting is called "Disable Indexer for performance on slow systems" in the "Options\ Preferences\ General" menu.
Tried it, not ideal.

After selecting the option and clicking OK, TB froze. Apparently, it can't stop the indexing once it's in full swing. Maybe it would have done it once the indexing finished, but I didn't bother to wait and End Tasked TB.

The second time round I was quick to do it after starting TB, and succeeded.

Also, I found the appropriate registry key: HKEY_CURRENT_USER\SOFTWARE\RIT\The Bat!\DisableSQLIndexer. I advise everyone having these problems to manually set this value to 1 in the Registry Editor before running v11.0.3.1.
 
 
 
#24
0  
Quote
Daniel van Rooijen wrote:
Have they explained how long this indexing should ordinarily run?
No, they just informed me about the new version 11.0.3.1 and the option to disable the indexing.

Quote
Daniel van Rooijen wrote:
Does it continue to run indefinitely even when no new mail comes in and you do not move, copy or delete any messages?
For me it stops after 10-15 minutes and I do not have a slow machine my any stretch of the imagination: i7 CPU, 16GB RAM, SSD... But for users with HDDs it's a nightmare lasting hours. And after each start of TB, the indexing just starts over.

Quote
Daniel van Rooijen wrote:
It doesn't make any sense to me, and rather than offering an option to disable the whole feature, I think they should figure out why it continues to run even after its goal (whatever that may be) has been achieved.
Agreed, but obviously they are not testing with large POP3 message bases. (See my post on disabling the indexing for more evidence.) As I wrote before, when I grab some time I will do some testing.

Quote
Daniel van Rooijen wrote:
Or, maybe the indexing is triggered every time that a small change has been made to the message base, but it is incredibly inefficient. Or, it doesn't update its indexes when a change is found, but it rebuilds them completely each time. Either way, Ritlabs seems to assume that this bug is a feature, and they're most likely wrong about that.
It is rebuilt each time TB starts. And I experienced new messages not making it into the threads until TB is restarted. This is simply appalling, exasperated by the fact that the existing indexing implementation can handle all of that quickly and reliably. In mere seconds for 100000 messages! Allocating several hundred MB of RAM.

Quote
Daniel van Rooijen wrote:
It might be helpful if they offer you an alpha version that enables tracing (maybe through an optional command line option - it would make the program even more unworkably slow), so each SQLite query that's run will also be recorded in a log file for bug hunting purposes.
Yes it would, but they haven't offered anything. However, for the main question I want to answer (is it a RAM allocation issue for SQLite), the regular version will do.
 
 
 
#25
0  
This changed nothing, latest v11.x with indexing off still makes my laptop unusable despite it is AMD Ryzen 7 5825U with SSD.  No pop3, just imap and TB! is a real crap now.
 
 
 
Pages: 1 2 Next