Paged queries with more than 100,000 matching objects
If a large number of objects match the query criteria, a paged query can consume a large amount of memory. If more than 100,000 objects match the query criteria, limit memory use by using multiple paged queries. Each paged query should retrieve results for no more than 100,000 objects. To do this, use the changeTimeMilliseconds
as the basis for generating the paged queries, as follows:
Procedure
Issue a request with a count entry value of zero and a
changeTimeMilliseconds
criterion with a range from zero to some time in the past, such as this:<queryRequest> <object> <query>+changeTimeMilliseconds:[0 TO 1262304000000.00] +retentionClass:hlthReg-107</query> <count>0</count> </object> </queryRequest>
If the
count
property in the response is greater than 100,000, repeat this step with an earlierchangeTimeMilliseconds
end time until thecount
property in the response is no more than 100,000.Use a paged query with:
- A
changeTimeMilliseconds
criterion that specifies the same time as you used in the last request in step 1 - A
count
entry value that specifies the number of objects you want per page - An
offset
entry that you increment by thecount
value in each request
<queryRequest> <object> <query>+changeTimeMilliseconds:[0 TO 1150000000000.00] +retentionClass:hlthReg-107</query> <sort>changeTimeMilliseconds</sort> <count>50</count> <offset>150</offset> </object> </queryRequest>
Stop when the code property of the status entry in the response is COMPLETE.- A
Repeat step 1 above using a
Use a curly opening brace for the range so that the last entry in the previous result set is not included in the new results. For example, use achangeTimeMilliseconds
entry that specifies a range with start value equal to the end value of thechangeTimeMilliseconds
range you used in step 2.changeTimeMilliseconds
value like this:changeTimeMilliseconds:{1150000000000.00 TO 1341000000000.00]
Then repeat step 2 using the new query criteria.
Repeat step 3 until you retrieve the last matching object.
Use a value of * (for an unlimited range) as the end of thechangeTimeMilliseconds
range in the last paged query to ensure that you retrieve all objects including those that were most recently added.