Recent changes to 412: After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Jeffrey Ratcliffe — Thu, 26 Jan 2023 18:39:26 -0000

It would be interesting to see whether they are left after scanimage uses the scanner. Can you try that?

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Wikinaut — Wed, 25 Jan 2023 22:15:49 -0000

Jeffrey, sorry, I meant, that eventually gscan2pdf is not properly shutting down a scan job it started, just an idea.

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Jeffrey Ratcliffe — Wed, 25 Jan 2023 20:21:47 -0000

As you point out yourself, those files are being created by the Brother driver, not gscanp2df, and not in the temporary directory that gscan2pdf uses. So I don't know how gscan2pdf should know that they can be directed. Perhaps it is worth filing a bug with Brother.

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Wikinaut — Wed, 25 Jan 2023 16:44:06 -0000

Further remark:

I noticed:

After having finished scanning of about 140 pages, properly saving as pdf and properly closing gscan2pdf - without crashes - I noticed in /tmp many files

"/tmp/brscan_jpeg_PAGEnn_hash"

which were apparently stemming from the scan process via SANE and the brother driver. I scanned with 600dpi/True Gray, the files are about 8-10 MB big.

I am telling you this, because perhaps gscan2pdf does not properly close/delete the the scan job/s - just as an idea what can be wrong. So in my case there are 1,4 GB of these brscan-files in /tmp for the last session of gscan2pdf (as said: without errors).

Is this a valuable information?

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Jeffrey Ratcliffe — Mon, 23 Jan 2023 16:55:00 -0000

No clue why changing ulimit didn't work. Perhaps it isn't supported everywhere.

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Wikinaut — Mon, 23 Jan 2023 11:36:22 -0000

Question: do you have an idea, why the(my) expressed setting of ulimit (to a value of 20000), why this did not fix the "too many open file issue"?

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Wikinaut — Sun, 22 Jan 2023 21:48:14 -0000

Uh, I always overlooked these to save options! Thanks for pointing out. (I used pdftk outside gscan2pdf, but using the built-in function is much more convenient, of course.)

Good luck by refactoring the code to overcome the "too many open files" problem - I am really looking forward.

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Jeffrey Ratcliffe — Sun, 22 Jan 2023 21:21:48 -0000

I would scan in batches of perhaps 100 pages and use the "append to PDF" option.

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Wikinaut — Sun, 22 Jan 2023 11:24:27 -0000

Thanks for your swift answer, however, I am in the need to scan dozens of data book for Computer History Museum/bitsavers.

Is there any quick workaround you could recommend for the time being?

Let me know, if I can help you.

PS. I do use the latest version gscan2pdf 2.13.2 on a modern Linux Mint system with 20 GB RAM.

#412 After scanning about 300 pages, gscan2pdf crashes with "too many open files" errors

Jeffrey Ratcliffe — Sun, 22 Jan 2023 10:56:37 -0000

I am in the process of completely rewriting how the pages are stored. This should solve this problem. However, this is not a trivial change and will take some time.