Page through large result sets

View as Markdown

One search call returns at most limit results. To go through all matching objects, sort the results by object ID and ask for the next page with an extra condition "object ID greater than the last ID so far" (keyset paging). Each page is an independent search, so objects added or removed between pages do not shift the pages.

Calls

SearchObjects, repeated, with:

Field Value
conditions Your conditions, plus status_value STATUS_TYPE_OBJECT_ID, CONDITION_TYPE_GREATER_THAN, integer = the last object ID of the previous page (0 for the first page)
sorting One SortRule with key -102: object ID, ascending
limit The page size

Stop when a page is empty or has_more_results is false.

Code

The builders (is_object_type, status_condition, integer, ...) are those of the search recipe.

Python

# Keyset paging: sort by object ID and ask each page for IDs above the last one seen.
last_id, page, total = 0, 0, 0
while True:
    # The next page: documents with IDs above the last one seen, in ID order.
    response = search.SearchObjects(IRPCSearch_pb2.SearchObjectsRequest(
        conditions=[structs.SearchConditionArray(value=[
            is_object_type(0),
            is_not_deleted(),
            status_condition(enums.STATUS_TYPE_OBJECT_ID,
                             enums.CONDITION_TYPE_GREATER_THAN, integer(last_id)),
        ])],
        sorting=structs.SearchResultsSorting(rules=structs.SortRuleValueArray(
            values=[structs.SortRule(key=SORT_BY_OBJECT_ID)])),
        limit=page_size))
    if not response.results:
        break

    # Count the page and remember its last ID for the next request.
    page += 1
    total += len(response.results)
    first_id = response.results[0].object.object_info.obj_id.item_id.internal_id
    last_id = response.results[-1].object.object_info.obj_id.item_id.internal_id
    print(f"Page {page}: {len(response.results)} documents, IDs {first_id}-{last_id}")

    # Stop after the last page.
    if not response.has_more_results:
        break

Full source: mfiles-grpc-python/examples/recipes/paging.py

C#

// Keyset paging: sort by object ID and ask each page for IDs above the last one seen.
int lastId = 0, page = 0, total = 0;
while (true)
{
    // The next page: documents with IDs above the last one seen. Stop if there are none.
    var request = new SearchObjectsRequest
    {
        Conditions =
        {
            new SearchConditionArray
            {
                Value =
                {
                    IsObjectType(0),
                    IsNotDeleted(),
                    StatusCondition(StatusType.ObjectId, ConditionType.GreaterThan, Integer(lastId)),
                },
            },
        },
        Sorting = new SearchResultsSorting
        {
            Rules = new SortRuleValueArray { Values = { new SortRule { Key = SortByObjectId } } },
        },
        Limit = (uint)pageSize,
    };
    var response = await search.SearchObjectsAsync(request);
    if (response.Results.Count == 0)
        break;

    // Count the page and remember its last ID for the next request.
    page++;
    total += response.Results.Count;
    int firstId = response.Results[0].Object.ObjectInfo.ObjId.ItemId.InternalId;
    lastId = response.Results[^1].Object.ObjectInfo.ObjId.ItemId.InternalId;
    Console.WriteLine($"Page {page}: {response.Results.Count} documents, IDs {firstId}-{lastId}");

    // A page that the limit did not cut short is the last one.
    if (!response.HasMoreResults)
        break;
}

Full source: mfiles-grpc-dotnet/examples/Recipes/Paging.cs

Go

// Keyset paging: sort by object ID and ask each page for IDs above the last one seen.
var lastID, page, total int32
for {
	response, err := search.SearchObjects(ctx, &mfiles.SearchObjectsRequest{
		Conditions: mfilesclient.AllOf(
			mfilesclient.IsObjectType(0),
			mfilesclient.IsNotDeleted(),
			mfilesclient.StatusCondition(mfiles.StatusType_STATUS_TYPE_OBJECT_ID,
				mfiles.ConditionType_CONDITION_TYPE_GREATER_THAN, mfilesclient.Integer(lastID)),
		),
		Sorting: mfilesclient.SortBy(sortByObjectID, false),
		Limit:   uint32(pageSize),
	})
	if err != nil {
		return err
	}
	results := response.GetResults()
	if len(results) == 0 {
		break
	}

	// Count the page; its last ID is where the next page starts.
	page++
	total += int32(len(results))
	firstID := objectID(results[0])
	lastID = objectID(results[len(results)-1])
	fmt.Printf("Page %d: %d documents, IDs %d-%d\n", page, len(results), firstID, lastID)

	// A page that says no more results follow is the last one.
	if !response.GetHasMoreResults() {
		break
	}
}

Full source: mfiles-grpc-go/examples/recipes/paging/main.go

Output

Output against the Sample Vault:

Page 1: 100 documents, IDs 23-236
Page 2: 100 documents, IDs 237-414
Page 3: 26 documents, IDs 415-445
Total: 226 documents in 3 pages

Pitfalls

See also