Ask HN: How to return large response over an API?

2 points

5 years ago

I am using a third party service which would transcribe an audio file and give me the entire transcription in a json format, I would read this and make a new json file by transforming it and send to s3. I am using python/django for it along with json, requests and boto3 module, however I noticed as the application kept running the memory consumption kept increasing.

To fix this I had to write it as a streaming response to a json file and use ijson (https://pypi.org/project/ijson/) to read the content which decreased memory utilisation a lot. So while looking around, I found many people had the same issue, so this got me thinking how would I have sent this content if i had built the third party api myself.

Some previous questions asked on stackoverflow,

https://stackoverflow.com/questions/2400643/is-there-a-memory-efficient-and-fast-way-to-load-big-json-files-in-python

https://stackoverflow.com/questions/11057712/huge-memory-usage-of-pythons-json-module

How do you send a response via an api for very large content, along with their advantages/disadvantages.

9 comments