Search

Introduction
Registering a source
Deciding who sees a hit
Searching without an engine
Narrowing a search to a place
Building documents from items
Adding a field to another feature’s documents
Describing fields
Adding facets and sorts
Making a source public
Writing a search feature
Advanced
See also

Introduction

Search in SPLENT has two sides. A feature that owns material, such as notes or pages, says what can be searched and who may see each hit. A search feature, with an engine or without one, builds the index and answers the reader. Neither imports the other, so a product can have either one, both or neither.

from flask import url_for

from splent_framework.db import db
from splent_framework.search import register_search_source


def init_feature(app):
    register_search_source("notes", "Notes", fetch=note_documents, resolve=note_result)


def note_documents():
    for note in Note.query.all():
        yield {"id": str(note.id), "title": note.title, "body": note.content}


def note_result(doc_id, user):
    note = db.session.get(Note, int(doc_id))
    if note is None or not notes_service.can_read(user, note):
        return None
    return {"title": note.title, "url": url_for("notes.show", note_id=note.id)}

Registering a source

A source is a key, a label and two functions.

fetch() yields every document, for building an index. A document is a dict with id, title and body, and may carry url, scope and any other field you want stored.

resolve(doc_id, user) turns one hit into a result this reader may see, with at least title and url, and usually snippet. It answers None to hide the hit.

The key names the index a search feature builds, and it is stored on every hit, so do not change it once something is indexed. The label is shown as the heading of your results. Pass it untranslated, or as a lazy string, since whoever shows it translates it.

Deciding who sees a hit

A document never says who may read it. An index is always a little behind, and a hit for a page that was just withdrawn must not show even its title. So the search feature asks resolve about every candidate, at the moment of the search.

Answering a whole page of candidates at once is faster. Add resolve_many.

def note_results(doc_ids, user):
    notes = Note.query.filter(Note.id.in_([int(i) for i in doc_ids])).all()
    return {
        str(note.id): {"title": note.title, "url": url_for("notes.show", note_id=note.id)}
        for note in notes
        if notes_service.can_read(user, note)
    }


register_search_source(
    "notes", "Notes", fetch=note_documents, resolve=note_result, resolve_many=note_results
)

Leave out what the reader may not see. Everything is denied by default, so an unknown source, a resolver that raises or an answer that is not a dict drops the candidates instead of showing them.

Searching without an engine

def find_notes(term, user, scope=None):
    notes = Note.query.filter(Note.title.ilike(f"%{term}%")).limit(20)
    results = (note_result(str(note.id), user) for note in notes)
    return [result for result in results if result]


register_search_source(
    "notes", "Notes", fetch=note_documents, resolve=note_result, find=find_notes
)

find(term, user, scope=None) answers a search when the product has no engine, or its engine is down. It returns results of the same shape as resolve. A source without find simply adds nothing then.

When a scope arrives, narrow the answer to it. With no index in between, nothing else stands between the reader and the whole site.

Narrowing a search to a place

def note_scopes(user):
    return [{"key": f"folder-{f.id}", "label": f.name} for f in folders_for(user)]


def current_note_scope():
    folder_id = request.view_args.get("folder_id") if request.view_args else None
    return f"folder-{folder_id}" if folder_id else None


register_search_source(
    "notes",
    "Notes",
    fetch=note_documents,
    resolve=note_result,
    scopes=note_scopes,
    current_scope=current_note_scope,
)

scopes(user) lists the places a reader may narrow to, already filtered to what that reader may see. current_scope() says where the reader is standing now, read from the request, or None when the page is not yours. A document says which place it belongs to in its scope field.

That way a search box can offer “search this folder” without the search feature knowing what a folder is.

Building documents from items

register_search_source(
    "notes",
    "Notes",
    fetch=None,
    resolve=note_result,
    items=lambda: Note.query.all(),
    document=lambda note: {"id": str(note.id), "title": note.title, "body": note.content},
)

With items and document instead of finished dicts, the framework builds each document. document(item) may answer None for an item that is not indexed.

This also lets another feature add a field to your documents.

Adding a field to another feature’s documents

from splent_framework.search import integer, register_search_field

register_search_field(
    "notes",
    "comments",
    integer(),
    batch=lambda notes: comment_counts([n.id for n in notes]),
)

The feature that knows a value adds it to documents it does not own. value(item) gives it for one item, and batch(items) for a list at once, in the same order, so indexing thousands of items does not query once per item.

It reaches only sources registered with items and document. A contributor that fails loses its own field for that batch, and the rest of the index is built.

Describing fields

from splent_framework.search import integer, keyword, nested, text

register_search_source(
    "notes",
    "Notes",
    fetch=None,
    resolve=note_result,
    items=all_notes,
    document=note_document,
    fields={
        "title": text(boost=3, suggest=True),
        "body": text(),
        "tags": keyword(text=True),
        "authors": nested(name=text(), orcid=keyword()),
        "words": integer(),
    },
)

fields says how each part of a document is matched and narrowed, in words no engine owns. A search feature turns them into its engine’s mapping. A field you do not name is stored, not searched.

Field	Means
`text(boost=1.0, suggest=False, parts=False)`	Words a reader types are looked for in it. `suggest` matches while the reader is still typing a word, and `parts` splits a name at its underscores, dots and dashes
`keyword(text=False, boost=1.0)`	A value narrowed by exactly, such as a tag. With `text=True` it is also matched by the words typed
`nested(**properties)`	A list of objects, each with the fields given, such as authors
`integer()`, `number()`, `date()`, `boolean()`	Values to narrow and sort by

from splent_framework.search import TERMS, register_search_facet, register_search_sort

register_search_facet("tags", "Tags", field="tags", kind=TERMS, sources=("notes",), order=10)
register_search_facet("words", "Length", field="words", sources=("notes",), order=20)

register_search_sort("newest", "Newest first", ("created_at", "desc"), order=10)
register_search_sort("best", "Best match", ("_score", "desc"), order=0)

A facet is what a reader narrows by, with how many results each choice leaves. A sort is an order the reader picks. _score is the best match.

Facets and sorts are registered by key, apart from the sources, so a feature can add a facet over a field it contributed to someone else’s documents.

Facet kind	For
`TERMS`	Values, several at once, such as tags. `size` caps how many are listed, `labels` names stored codes
`RANGE`	A least and a most. The default
`LOG_RANGE`	Orders of magnitude, for numbers that span many
`BOOLEAN_FACET`	Yes or no
`DATE_FACET`	A year, or from one day to another

group gathers facets under a heading, order places them, lower first, and help explains one to the reader.

Making a source public

register_search_source(
    "notes",
    "Notes",
    fetch=None,
    resolve=note_result,
    items=published_notes,
    document=note_document,
    public=True,
    present=lambda doc: {**doc, "url": external_url_for("notes.show", note_id=doc["id"])},
    card="notes/search_card.html",
)

When everything a source indexes may be read by anyone, mark it public=True. A search feature may then answer from the index itself, with exact counts per facet, instead of asking resolve about every hit.

present(document) says what a page or an API shows of one result read from the index. card is a template that draws one result of this source.

Writing a search feature

A search feature reads the registry and never imports the features that filled it.

from splent_framework.search import get_search_sources, search_documents
from splent_framework.search.search_registry import resolve_search_hits

for source in get_search_sources():
    engine.index(source["key"], search_documents(source["key"]))

ids = engine.query("notes", term)
shown = resolve_search_hits("notes", ids, current_user)
results = [shown[i] for i in ids if i in shown]

Walk the engine’s ids and look each one up, so you keep its ranking and show only what survived.

Function	Answers
`get_search_sources()`, `get_search_source(key)`	The sources, in display order, or one of them
`search_documents(key, items=None)`, `search_document(key, item)`	Documents ready to index, with the fields others added
`get_search_fields(key)`	A source’s fields, its own and the added ones
`get_search_facets(source_key=None)`, `get_search_sorts()`	Facets and sorts, in order
`resolve_search_hits(key, doc_ids, user)`	`{doc_id: result}` of what this reader may see
`get_search_scopes(user)`, `get_current_scope()`	The places to narrow to, and where the reader stands
`present_search_result(key, document)`	What to show of one hit of a public source

resolve_search_hits, get_search_scopes and get_current_scope live in splent_framework.search.search_registry. Import the rest from splent_framework.search.

Advanced

The registry is filled once, while the features load, and only read while serving. It is emptied when an app starts, so a product sees only what its own features registered.