Repository navigation
Performance Issue with Extracting genres Attribute from Media #1449
Description
Activity
See responses in #713, #1007, #1128, and #1188 regarding partial objects.
And
autoreloadconfiguration option.https://python-plexapi.readthedocs.io/en/latest/configuration.html#section-plexapi-options
Thank you for your reply. I completely understand that every time I call the
mediaproperty, it requests all fields of that media from the Plex backend again. This is indeed a very good handling logic, and I agree with it.However, I still have some concerns about performance. The reason is that there is a significant difference between this approach and directly sending requests to PMS.
Assuming I need to get the latest metadata for all media, I compared the following:
- Test1: Iterate through the
mediaslist and call themedia.reload()method for eachmedia. - Test2: Iterate through the
mediaslist and construct the URL usingf"{baseurl}{media.__dict__.get(key)}"for eachmedia, then initiate a network request. - Test3: Run Test1 using 8 threads.
- Test4: Run Test2 using 8 threads.
The results are:
INFO:root:msg="Media count: 802" INFO:root:msg="Test1" duration=84150ms INFO:root:msg="Test2" duration=5533ms INFO:root:msg="Test3" duration=33943ms INFO:root:msg="Test4" duration=1296msMy test code:
import logging import time import concurrent.futures import requests from plexapi.myplex import PlexServer logging.basicConfig(level=logging.INFO) def threads(datalist, func, thread_count): def chunks(lst, n): for i in range(0, len(lst), n): yield lst[i:i + n] chunk_size = (len(datalist) + thread_count - 1) // thread_count list_chunks = list(chunks(datalist, chunk_size)) with concurrent.futures.ThreadPoolExecutor(max_workers=thread_count) as executor: result_items = list(executor.map(func, [item for chunk in list_chunks for item in chunk])) return result_items def test(baseurl, token): client = PlexServer(baseurl, token) medias: list = client.library.all(libtype='movie') logging.info(f'msg="Media count: {len(medias)}"') t1 = int(time.time() * 1000) # test1 for media in medias: metadata = media.reload().__dict__ logging.debug(metadata) t2 = int(time.time() * 1000) logging.info(f'msg="Test1" duration={(t2 - t1)}ms') # test2 for media in medias: metadata = requests.get( url=f'{baseurl}{media.__dict__.get('key')}', headers={'X-Plex-Token': token, 'Accept': 'application/json'} ).json().get("MediaContainer", {}).get("Metadata", [{}])[0] logging.debug(metadata) t3 = int(time.time() * 1000) logging.info(f'msg="Test2" duration={(t3 - t2)}ms') # test3 def query_metadata_1(_media): _metadata = _media.reload().__dict__ logging.debug(_metadata) threads(medias, query_metadata_1, 8) t4 = int(time.time() * 1000) logging.info(f'msg="Test3" duration={(t4 - t3)}ms') # test3 def query_metadata_2(_media): _metadata = requests.get( url=f'{baseurl}{_media.__dict__.get('key')}', headers={'X-Plex-Token': token, 'Accept': 'application/json'} ).json().get("MediaContainer", {}).get("Metadata", [{}])[0] logging.debug(_metadata) threads(medias, query_metadata_2, 8) t5 = int(time.time() * 1000) logging.info(f'msg="Test4" duration={(t5 - t4)}ms') if __name__ == '__main__': test('http://127.0.0.1:32400', 'MyPlexToken')- Test1: Iterate through the
Your direct query doesn't include all the URL parameters, so you are missing a lot of metadata.
A more apples-to-apples comparison would be
metadata = media.reload( checkFiles=False, includeAllConcerts=False, includeBandwidths=False, includeChapters=False, includeChildren=False, includeConcerts=False, includeExternalMedia=False, includeExtras=False, includeFields=False, includeGeolocation=False, includeLoudnessRamps=False, includeMarkers=False, includeOnDeck=False, includePopularLeaves=False, includePreferences=False, includeRelated=False, includeRelatedCount=False, includeReviews=False, includeStations=False ).__dict__
vs.
metadata = requests.get( url=f'{baseurl}{media.__dict__.get("key")}', headers={'X-Plex-Token': token, 'Accept': 'application/json'} ).json().get("MediaContainer", {}).get("Metadata", [{}])[0]
Or
metadata = media.reload().__dict__
vs.
metadata = requests.get( url=f'{baseurl}{media.__dict__.get("key")}', headers={'X-Plex-Token': token, 'Accept': 'application/json'}, params=media._INCLUDES ).json().get("MediaContainer", {}).get("Metadata", [{}])[0]
The remaining difference in performance will be the small overhead for building the
PlexObjectinstead of just returning raw xml/json.Understood. You are right; based on my tests, querying more metadata significantly increases the overhead. In a single-threaded environment, directly requesting the JSON is only 16ms faster on average per media item compared to constructing an object. However, with 8 threads, the difference is negligible.
For my use case, I indeed do not need that much metadata. However, using
media.reload( checkFiles=False, includeAllConcerts=False, includeBandwidths=False, includeChapters=False, includeChildren=False, includeConcerts=False, includeExternalMedia=False, includeExtras=False, includeFields=False, includeGeolocation=False, includeLoudnessRamps=False, includeMarkers=False, includeOnDeck=False, includePopularLeaves=False, includePreferences=False, includeRelated=False, includeRelatedCount=False, includeReviews=False, includeStations=False )seems overly cumbersome. But my problem is resolved. Thank you.
The biggest contributor is probably just
checkFiles=Falsebecause then Plex won't scan your hard drive to check the file exists. The other include parameters should have very small impact on the speed.I think I will change the default to
checkFiles=Falseand have the user include it explicitly if they need to check for the file.I think the solution of changing the default values is quite good.
(However, in my use case, I ultimately opted for the approach of concatenating the URL to send network requests and parse the JSON. Because even if there are only slight differences in a single request, when there are many items in the media library, the total time quickly expands from a few seconds to over a minute. Of course, perhaps it is completely unnecessary to be so obsessed with performance optimization. A friend of mine said that people really don't care how long a scheduled script takes to run. I think he is right; in fact, when I'm actually using it, I don't care about the runtime either. It's just that during the process of writing the script, I still feel troubled by it.)
Describe the Bug
For the following code snippet, I observed a significant performance discrepancy:
The output is:
This is horrifying.
Code Snippets
No response
Expected Behavior
No response
Additional Context
No response
Operating System and Version
windows server 2022
Plex Media Server Version
1.40.4.8679
Python Version
3.12
PlexAPI Version
4.15.15