Is FPDF2 faster than FPDF1?

[simkessy]

I can't seem to find speed comparisons. I'm looking at some old code that is using fpdf==1.7.3 and we have a particularly large PDF generation that is taking over 6 hours. Our jobs are currently set to cancel after 6 hours so I haven't been able to successfully generate this PDF.

I'm wondering if upgrading to FPDF2 will help with performance. These PDFs have can have hundreds if not thousands of images so they become quite large. Are there any suggestions on how to deal with this?

I was considering using multiprocessing to create pages, is this a viable solution to speeding up or reducing the memory consumption?

Thank you

[Lucas-C]

Hi @simkessy!

One of the main reasons why I contributed to fpdf2 in the begining was to make it faster than PyFPDF:
reingart#164

If you want to compare specific use case, you could make your own benchmark.
You can take inspiration from this recent script that I made:
#742 (comment)

I've wondering if upgrading to the latest FPDF2 will help with performance.

I bet it will 😊

These PDFs have can have hundreds if not thousands of images so they become quite large. Are there any suggestions on how to deal with this?

Not really, but you should try to measure where the bottlneck is in terms of processing time,
that is what rendering step takes the most tim to complete.
In my experience, it is usually in the call to FPDF.output().
Is it really the PDF-rendering part of your job that takes most of those 6 hours??

I was considering using multiprocessing to create pages, is this a viable solution to speeding up or reducing the memory consumption in FPDF2?

I'm not sure that this approach would be helpful...
But what do you have in mind exactly? What parts of the rendering would you perform in parallel?

If you make the switch to fpdf2, I'd be curious to know how much gain you get!
And I can also try to provide more advice once you use fpdf2, to optimize your job, but I will need a bit more information on what it does.
For example, as you mentioned many images, you could consider image downscaling, which could reduce the end PDF document size and reduce your job memory usage: https://pyfpdf.github.io/fpdf2/Images.html#oversized-images-detection-downscaling

[simkessy]

I'm not sure that this approach would be helpful...
But what do you have in mind exactly? What parts of the rendering would you perform in parallel?

So we loop over an array of images that we retrieve from s3, essentially the flow is: Loop over array > for each item make a s3 read > get image as string and convert to bytes using boto3 > pdf.add_page > pdf.image

I was wondering if all that could be a parallel operation where we generate a page / image, then combine them to create the output.

Also thanks for the earlier response, we'll definitely upgrade and do some profiling now.

[Lucas-C]

I think an approach where you parallelize the images fetching from S3 to memory, and after that build the PDF, would make things faster while keeping the code complexity relatively low.
fpdf2 is really not designed to allow for several pages to be built in parallel, in distinct processes or threads.

To parallelize HTTP calls, you could use aiohttp ( cf. https://medium.com/@cgarciae/making-an-infinite-number-of-requests-with-python-aiohttp-pypeln-3a552b97dc95), gevent, multiprocessing.pool.ThreadPool or any other approach.

[gmischler]

So we loop over an array of images that we retrieve from s3, essentially the flow is: Loop over array > for each item make a s3 read > get image as string and convert to bytes using boto3 > pdf.add_page > pdf.image

Have you done any verification to see where in this pipeline the actual bottleneck is located?
Same as @Lucas-C , I suspect that fetching the data from S3 would take the most time, and parallelizing that would give you the largest gains.

If there is a need to increase the performance of fpdf2 itself, then memory consumption might be the main issue. It needs to hold all the (uncompressed!) data of the output file in memory, after all. A possible solution in this case could be to generate seperate PDFs for maybe 100 images/pages at a time, and combine them with a simpler tool like pdfrw in the end.

But you really need to profile your actual setup before changing anything beyond the migration from fpdf to fpdf2. With the amount of data you're dealing with, bottlenecks can sneak up on you in the most unexpected places.

[Lucas-C]

I'm going to keep this issue open for now, waiting some feedbacks on your experiments @simkessy 😊
And if it stays inactive for a a few weeks I'll close it then.

[simkessy]

I'm going to keep this issue open for now, waiting some feedbacks on your experiments @simkessy 😊
And if it stays inactive for a a few weeks I'll close it then.

I got the upgrade working and wow, it changed the pdf generation time from 6+ hours (not sure since job would hit thresholds and fail) to 20 minutes generating a PDF with 130K pages and about 550mb, running on a machine significantly more underpowered than where we run our jobs.

I am also using aws s3 sync to download all necessary files first, I'm not sure how much of an improvement that is, im sure it's not as much as just updating to fpdf2.

Anyways, thanks for this library and your help @Lucas-C

[Lucas-C]

Thank you very much for this feedback @simkessy!

Glad that fpdf2 worked out that well for your use-case ^^

I'm closing this issue now, but feel free to open other discussions & issues on this project if you need to!