Let's review how Aerogramme performances became more predictable, why it's important, and showcase how user testing helped surfacing bugs.
This minor version of Aerogramme put the focus on 2 aspects of the software: predictable performances & collecting user feedbacks. In the following, I describe both aspect in details.
Improving predictability
In the previous blog post, we asked ourselves Does Aerogramme use too much memory?. From the discussion, we surfaced it was not acceptable that some specific queries (FETCH FULL or SEARCH TEXT) where loading the full mailbox in memory. It's concerning as the mail server will be used by multiple users and as a limited amount of resources, so we don't want a user allocating all the memory. In other words, we want to have a per-user resource usage that remain as stable as possible. These ideas are developed more in depth in the Amazon article Reliability & Constant Work.
As part of the conclusion, we identified that streaming emails content would solve our problem.
In practice, I rewrote the relevant part of the code to return a futures::stream::Stream instead of a Vec.
Then, for both requests, I re-run the same benchmark to have a before/after comparison for both commands.
Next, the first pictures is for FETCH FULL, the second one for SEARCH TEXT.
For both FETCH and SEARCH, the changes are identical. Before, the command was executed in ~5 seconds, allocated up to 300MB of RAM, and used up to 150% of CPU. After, the command took ~30 to get executed, allocated up to 400MB of RAM, used up to 40MB of RAM, and used up to 80% of CPU sporadically.
Here is why it's a positive thing: now the memory consumption of Aerogramme is capped approximately by the biggest email accepted by your system (25MB, multiplied by a small constant).
It has also other benefits: it prevents the file descriptor and other network ressource exhaustion, and it adds fairness between users. Indeed, a user can't monopolize all the ressources of the servers (CPU, I/O, etc.) anymore, and thus multiple users requests are thus intertwined by the server (we assume the number of users is greatly superior to the number of cores). And again, it leads to better predictability, as per-user requests completion will be less impacted by other requests.
Another concern is the RAM consumption with the IDLE feature. The real cause is that we retain in-memory the full user profile (mailbox data, IO connectors, etc.): we should instead keep only the minimum data to be waken-up. That's the ideal fix, the final solution, that would take lots of time to design and implement. This fix is not necessary now, instead we can simply try to optimize the size of a full user profile in memory. On this aspect, the aws-sdk-s3 crate has the following note:
Client construction is expensive due to connection thread pool initialization, and should be done once at application start-up.
Digging deeper in the crate dependencies, we learn from the aws-smithy-runtime crate, we can read:
[Constructing] a Hyper client with the default TLS implementation (rustls) [...] can be useful when you want to share a Hyper connector between multiple generated Smithy clients.
It seems to be exactly what we want to do, but to be effective, we must do the same thing for K2V. After implementing these features, I got the following plot:
First, the spikes are more spaced because I am typing the command by hands, not because Aerogramme is slower! Another non relevant artifact is the end of the memory plot: memory is not released on the left plot as I cut the recording before typing the LOGOUT command. What's interesting is the memory usage range: on the left, it's ~20MB, on the right it's ~10MB. By sharing the HTTP Client, we thus use twice less memory per-user, down to around ~650kB/user for Aerogramme 0.2.2 Back to our 1k users, we get 650MB of RAM, 6.5GB for 10k, and thus 65GB for 100k. So in theory, it seems OK, but our sample and our methodology is too dubious to confirm that in practice such memory usage will be observed.
In the end, I think for now IDLE RAM usage in Aerogramme is acceptable, and thus we can move on other aspects without fearing that IDLE will make the software unusable.
Users feedbacks
Dovecot AUTH continuation inlining - When a username + password is short, the Dovecot SASL Auth protocol allows the client (here Postfix) to send the base64 inlined, without having to wait for the continuation. It was not supported by Aerogramme and was preventing some users from authenticating.
Pipelining limits (reported by Nicolas) - Pipeling limit set to 3. Avoiding DoS resources. But failing some honest clients like Mutt. Bumped to 64, will be watched in the next months.
SASL Auth subtleties (reported by Nicolas) - Authorization can be empty, or can be set to the same value as Authentication. Second case not handled but required by Fair Email (thx Nicolas)
Thunderbird Autodiscovery issues (reported by LX & Nicolas) - K9 stable does not support the %EMAILLOCALPART% placeholder.
K9 beta (6.714) does not support some values marked as obsolete in the authentication field: plain is not supported anymore, password-cleartext must be used instead.
Content-Type is important also, if a wrong one is sent, content is silently ignored by some clients.
Today autodiscovery is not part of Aerogramme, but due to these issues, and given that generating these files
on the fly could improve compatibility (email is passed a query parameter), so we could directly put the right username
instead of relying on placeholders that are not well supported.
Broken LITERAL+ (reported by Maxime) - It was not possible to copy more than one email at once to an Aerogramme mailbox. It was due to the fact we were using an old version of imap-flow that was not correctly supporting LITERAL+. Upgrading imap-flow to the latest version fixed the problem.
Broken IDLE (reported by Maxime) - After updating imap-flow, we started noticing some timeouts in Thunderbird due to IDLE bugs. When IDLE was implemented in Aerogramme, the code was not ready in imap-flow, and thus I used some hacks. But upgrading the library broke my hacks for the best: now imap-flow supports IDLE out of the box, and thus Aerogramme code is now cleaner and more maintainable.
Some others quality of life feedbacks not reported here have been made by MrFlos & Nicolas, thanks to all the people that took part in this debugging adventure.
Conclusion: download and test this new version!
Do not get me wrong: Aerogramme is still not ready for prime time. But by operating it for real, we start understanding better how it behaves, what are the rough edges, what we want to improve, what we need, etc.
If you are interested in being an Aerogramme early adopter, it might be the good time to setup a test cluster, try Aerogramme, share your use cases, in order to shape its future and be ready to go in production when its public beta will be announced.
docker run --net host registry.deuxfleurs.org/aerogramme:0.2.2
It will launch Aerogramme in development mode. Username is alice, password is hunter2, email is alice@example.tld.