Page through large result sets

View as Markdown

One search call returns at most limit results. To go through all matching objects, sort the results by object ID and ask for the next page with an extra condition "object ID greater than the last ID so far" (keyset paging). Each page is an independent search, so objects added or removed between pages do not shift the pages.

Calls

SearchObjects, repeated, with:

Field Value
conditions Your conditions, plus status_value STATUS_TYPE_OBJECT_ID, CONDITION_TYPE_GREATER_THAN, integer = the last object ID of the previous page (0 for the first page)
sorting One SortRule with key -102: object ID, ascending
limit The page size

Stop when a page is empty or has_more_results is false.

Code

The builders (is_object_type, status_condition, integer, ...) are those of the search recipe.

Python

# Keyset paging: sort by object ID and ask each page for IDs above the last one seen.
last_id, page, total = 0, 0, 0
while True:
    response = search.SearchObjects(IRPCSearch_pb2.SearchObjectsRequest(
        conditions=[structs.SearchConditionArray(value=[
            is_object_type(0),
            is_not_deleted(),
            status_condition(enums.STATUS_TYPE_OBJECT_ID,
                             enums.CONDITION_TYPE_GREATER_THAN, integer(last_id)),
        ])],
        sorting=structs.SearchResultsSorting(rules=structs.SortRuleValueArray(
            values=[structs.SortRule(key=SORT_BY_OBJECT_ID)])),
        limit=page_size))
    if not response.results:
        break

    page += 1
    total += len(response.results)
    first_id = response.results[0].object.object_info.obj_id.item_id.internal_id
    last_id = response.results[-1].object.object_info.obj_id.item_id.internal_id
    print(f"Page {page}: {len(response.results)} documents, IDs {first_id}-{last_id}")

    if not response.has_more_results:
        break

Source: python/recipes/paging.py

C#

// Keyset paging: sort by object ID and ask each page for IDs above the last one seen.
int lastId = 0, page = 0, total = 0;
while (true)
{
    var request = new SearchObjectsRequest
    {
        Conditions =
        {
            new SearchConditionArray
            {
                Value =
                {
                    IsObjectType(0),
                    IsNotDeleted(),
                    StatusCondition(StatusType.ObjectId, ConditionType.GreaterThan, Integer(lastId)),
                },
            },
        },
        Sorting = new SearchResultsSorting
        {
            Rules = new SortRuleValueArray { Values = { new SortRule { Key = SortByObjectId } } },
        },
        Limit = (uint)pageSize,
    };
    var response = await search.SearchObjectsAsync(request);
    if (response.Results.Count == 0)
        break;

    page++;
    total += response.Results.Count;
    int firstId = response.Results[0].Object.ObjectInfo.ObjId.ItemId.InternalId;
    lastId = response.Results[^1].Object.ObjectInfo.ObjId.ItemId.InternalId;
    Console.WriteLine($"Page {page}: {response.Results.Count} documents, IDs {firstId}-{lastId}");

    if (!response.HasMoreResults)
        break;
}

Source: csharp/Recipes/Paging.cs

Go

// Keyset paging: sort by object ID and ask each page for IDs above the last one seen.
var lastID, page, total int32
for {
	response, err := search.SearchObjects(ctx, &mfiles.SearchObjectsRequest{
		Conditions: mfilesclient.AllOf(
			mfilesclient.IsObjectType(0),
			mfilesclient.IsNotDeleted(),
			mfilesclient.StatusCondition(mfiles.StatusType_STATUS_TYPE_OBJECT_ID,
				mfiles.ConditionType_CONDITION_TYPE_GREATER_THAN, mfilesclient.Integer(lastID)),
		),
		Sorting: mfilesclient.SortBy(sortByObjectID, false),
		Limit:   uint32(pageSize),
	})
	if err != nil {
		return err
	}
	results := response.GetResults()
	if len(results) == 0 {
		break
	}

	page++
	total += int32(len(results))
	firstID := objectID(results[0])
	lastID = objectID(results[len(results)-1])
	fmt.Printf("Page %d: %d documents, IDs %d-%d\n", page, len(results), firstID, lastID)

	if !response.GetHasMoreResults() {
		break
	}
}

Source: go/recipes/paging/main.go

Output against the Sample Vault:

Page 1: 100 documents, IDs 23-236
Page 2: 100 documents, IDs 237-414
Page 3: 26 documents, IDs 415-445
Total: 226 documents in 3 pages

Pitfalls

See also