With MS Dynamics CRM 2011 or later (both online and locally installed) you could get an error message saying "Paging cookie is required when trying to retrieve a set of records on any high pages.". You would get this error message when trying to read more than 5000 records from CRM. This has now been fixed.
If you are using RapidiOnline to integrate with MS Dynamics CRM, you can skip the following details and just jump to the upgrade information at the end. Take me to the upgrade section
Microsoft Dynamics CRM would return a TotalCount of 5000 no matter how many records there were in the table. We already had implemented reading of just 200 records per request and sending the PagingCookie previously returned by CRM back with each request also. This worked fine for the first 4800 records read from CRM, but as soon as you passed the 5000 records mark, you would get the above error.
After a lot of debugging, we found out that it did not matter what you set in PagingCookie as long as you were not retrieving records after record number 5000 in the table. You would not get any error from CRM - CRM just ignores any PagingCookie sent (but still correctly returns the 200 next records you ask for). Very strange indeed !
The exact format of the PagingCookie is not documented by Microsoft, so it took some more trial and error to finally find the correct format for the PagingCookie. It has to be sent exactly as it should be - down to using the exact case of the words and quoting and html escaping the string in the correct places.
How to solve the issue
Below you see the raw xml example of a correct PageInfo and PagingCookie for the MS Dynamics CRM 2011 API:
<a:PageInfo>
<a:Count>200</a:Count><a:PageNumber>2</a:PageNumber><a:PagingCookie><cookie page="1"><productid last="{136576CA-2B05-E534-80F7-C4346BAC7BE8}" first="{D3A092A5-2B05-E534-80F7-C4346BAC7BE8}" /></cookie></a:PagingCookie><a:ReturnTotalRecordCount>true</a:ReturnTotalRecordCount></a:PageInfo>
For the first request (for PageNumber 1) you set PagingCookie to nil. CRM will then return a PagingCookie as part of the response. You then have to modify the PagingCookie to look like above (using < instead of < and so on) and send this new PagingCookie as part of the request for PageNumber 2. This will enable CRM to read past the 5000 record limit and read all records. I hope this makes it more clear and saves somebody from the trouble we been through figuring this out.
How to Upgrade
The PagingCookie is fixed in RapidiOnline version 3.2.91p. If you are using CRM online, you need to upgrade the central version of Rapidi, otherwise you need to upgrade the locally installed RapidiConnector for CRM.
Please contact our support to help you with these upgrades.
Thanks
Michael
