Use memory-mapped files in Snark

Opened 3 years ago

Last modified 3 years ago

#2281newenhancement

Use memory-mapped files in Snark

Reported by:Zlatin BalevskyOwned by:zzz Priority: minor Milestone: undecided Component: apps/i2psnark Version: 0.9.35 Keywords: nio Cc:

Parent Tickets:

Sensitive: no

Description

It makes sense to use memory-mapped files instead of RandomAccessFile because of performance.

Subtickets

changed milestone to %undecided

added #2281 apps i2psnark undecided labels

comment:1 Changed 3 years ago by zzz

Component:apps/other → apps/i2psnarkOwner: set to zzz

comment:2 Changed 3 years ago by zzz

you mean a nio FileChannel?? or what? link to article on performance?

comment:3 Changed 3 years ago by Zlatin Balevsky

​https://docs.oracle.com/javase/7/docs/api/java/nio/MappedByteBuffer.html

Here is a brief but good description why memory-mapped files outperform other types of disk i/o: ​https://stackoverflow.com/questions/192527/what-are-the-advantages-of-memory-mapped-files

Also ​https://www.codeproject.com/Tips/683614/10-Things-to-Know-about-Memory-Mapped-File-in-Java

comment:4 Changed 3 years ago by zzz

quote ​https://docs.oracle.com/javase/7/docs/api/java/nio/channels/FileChannel.html#map(java.nio.channels.FileChannel.MapMode,%20long,%20long -

For most operating systems, mapping a file into memory is more expensive than reading or writing a few tens of kilobytes of data via the usual read and write methods. From the standpoint of performance it is generally only worth mapping relatively large files into memory.

In snark, generally we don't do repeated reads/writes of a particular portion of a file. We read or write it and move on to something else. Snark can randomly read or write to one of thousands of files totalling gigabytes. And we have to make it work on everything from a low-memory RPi to a big server. I don't know how to manage that and implicit in the MappedByteBuffer? implementation is you're doing repeated reads/writes to the same region.

So mapping a portion of a file into memory seems like a bad fit for snark.

comment:5 Changed 3 years ago by Zlatin Balevsky

Two things:

1. There are repeated reads from the same file region when uploading the same piece multiple times.

2. Mapping a file to memory does not mean that the contents will actually get loaded into memory (although that can be forced, which is a bad idea). It just means that the OS may choose to load and unload various regions of the file depending on access patterns and available memory.

So the only situation where mapping is clearly bad idea is 32-bit platforms where the address space is only 4GB. On a 64-bit raspberry pi, even if it has only half a gig of real memory it is safe to map up to 264 bytes of files. It just means the OS will have to un-map more frequently.