https://www.baeldung.com/java-read-lines-large-file
The problem with this approach is that all the file lines are kept in memory – which will quickly lead to OutOfMemoryError if the File is large enough.
What's more, we usually don't need all of the lines in the file in memory at once – instead, we just need to be able to iterate through each one, do some processing and throw it away. So, this is exactly what we're going to do – iterate through the lines without holding all of them in memory.
The same can be achieved using the Commons IO library as well, by using the custom LineIterator provided by the library:
LineIterator it = FileUtils.lineIterator(theFile, "UTF-8");
try {
while (it.hasNext()) {
String line = it.nextLine();
// do something with line
}
} finally {
LineIterator.closeQuietly(it);
}
Since the entire file is not fully in memory – this will also result in pretty conservative memory consumption numbers
