Improve getEdit() performance a bit #2821

New issue

Closed

opened 2015-07-24 16:46:13 +00:00 by wjt · 4 comments

wjt commented

2015-07-24 16:46:13 +00:00

I (programmatically) made an static edit with 13k clips, and the request times out when I try to watch it :-) so I spent some quality time with a profiler. The vast majority of time is spent JSONifying all the annotations for each clip. I found a couple of easy fixes, patches attached.

With these patches, 98% of time is still spent in Edit.get_layers. Within that:

File: /srv/pandora/pandora/edit/models.py
Function: get_layers at line 503

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   503                                               def get_layers(self, user=None):
   ...
   525       648       111992    172.8     49.1              for a in qs.order_by('start').select_related('user'):
   526       632        86694    137.2     38.0                  ll.append(a.json(user=user))

That 38% in Annotation.json is mostly get_entity():

File: /srv/pandora/pandora/annotation/models.py
Function: json at line 216

Line #      Hits         Time  Per Hit   % Time  Line Contents
==============================================================
   216                                               def json(self, layer=False, keys=None, user=None):
   ...
   235        28        44561   1591.5     65.2                  entity = self.get_entity()
   ...
   274       632         6168      9.8      9.0          subtitles = get_by_key(settings.CONFIG['layers'], 'isSubtitles', True)

It's hard to see how to do much better without changing the Annotations schema to have a nullable reference to the entity. Maybe that's one for another time.

I (programmatically) made an static edit with 13k clips, and the request times out when I try to watch it :-) so I spent some quality time with a profiler. The *vast* majority of time is spent JSONifying all the annotations for each clip. I found a couple of easy fixes, patches attached. With these patches, 98% of time is still spent in Edit.get_layers. Within that: ``` File: /srv/pandora/pandora/edit/models.py Function: get_layers at line 503 Line # Hits Time Per Hit % Time Line Contents ============================================================== 503 def get_layers(self, user=None): ... 525 648 111992 172.8 49.1 for a in qs.order_by('start').select_related('user'): 526 632 86694 137.2 38.0 ll.append(a.json(user=user)) ``` That 38% in Annotation.json is mostly get_entity(): ``` File: /srv/pandora/pandora/annotation/models.py Function: json at line 216 Line # Hits Time Per Hit % Time Line Contents ============================================================== 216 def json(self, layer=False, keys=None, user=None): ... 235 28 44561 1591.5 65.2 entity = self.get_entity() ... 274 632 6168 9.8 9.0 subtitles = get_by_key(settings.CONFIG['layers'], 'isSubtitles', True) ``` It's hard to see how to do much better without changing the Annotations schema to have a nullable reference to the entity. Maybe that's one for another time.