With 17 layers and 12 clipLayers, this repeated fetching was around 49%
of the cost of this function, which was in turn 94% of the cost of
creating many new annotations with mostly-unique endpoints. This helps a
bit...
If the order of clipLayers is not meant to be significant to sortvalue
(which I assume it is) then this could be simpler.
The idea here is to have several layers which share a set of tags. This
mirrors what we already have if several layers reference the same type
of entity. You might have config like this:
{
"id": "keywords",
"title": "Keywords",
"canAddAnnotations": {"member": true, "staff": true, "admin": true},
"item": "Keyword",
"overlap": true,
"type": "string",
"autocomplete": true,
"autocompleteKeys": ["keywords", "minorkeywords"]
},
{
"id": "minorkeywords",
"title": "Minor Keywords",
"canAddAnnotations": {"member": true, "staff": true, "admin": true},
"item": "Keyword",
"overlap": true,
"type": "string",
"autocomplete": true,
"autocompleteKeys": ["keywords", "minorkeywords"]
},
Now, adding new keywords in either bin will offer autocompletions from
the union of the two layers. The other option would be to do this on the
server side, but I thought this was a less invasive way to achieve this.
The case must be correct anyway for the layer to be found in
settings.CONFIG['layers']. Running this:
Q(annotation__layer__iexact='foo') &
Q(annotation__findvalue__icontains='bar')
compiles to
upper(layer) = upper('foo') and
...
which can't use the case-sensitive annotation_annotation_layer index.
This:
Q(annotation__layer__exact='foo') &
Q(annotation__findvalue__icontains='bar')
can. (It still can't use the findvalue_like index, though! The other
option is to add indices on upper(layer) and upper(findvalue)
[varchar_pattern_ops].)
This has several benefits:
• Clip.get_layers() (used by smart edits) and Item.get_layers() pick up
the select_related('user') optimization added for static edits in
r5007.
• Static edits and items pick up the optimization from r4941 to select
annotations once, not once per layer.
Fetching an item with ~1000 annotations took ~1s without this patch,
~0.34s with this patch. Another item with ~6000 annotations took ~11.6s
before, ~8.6s after.
Because this block is moved out to the top:
if user and user.is_anonymous():
user = None
then, for anonymous users,
"editable": false,
is no longer included in the annotations. The old behaviour ended up
including this key in all layers listed before the first private layer
in the config, and leaving it out from later ones. So this new behaviour
is more consistent.
The expensive part of fetching an edit is JSONifying the clips'
annotations. Profiling showed that the main cost was Annotation.json(),
and within that:
File: /srv/pandora/pandora/annotation/models.py
Function: json at line 216
Line # Hits Time Per Hit % Time Line Contents
==============================================================
216 def json(self, layer=False, keys=None, user=None):
217 632 827 1.3 0.1 j = {
218 632 1048170 1658.5 89.6 'user': self.user.username,
219 }
Obviously this join just moves some of the cost further out, but it
brings my micro-benchmark down from 1.3s to 0.3s.
If (for example) you have a type of entities, "participants", and a
second annotation layer for them, like this:
{
"id": "participantsdiscussed",
"title": "Participants discussed",
"type": "entity",
"entity": "participants"
}
then we should be passing key="participants" to autocompleteEntities
(since that is the type of entity), not key="participantsdiscussed"
(which is not a type of entities and raises an error).
