We’re trying to export process mining data via REST API for analysis in our BI tools. The export works fine for small date ranges (1-2 weeks), but when we try to pull 3+ months of data, the API times out with a 504 Gateway Timeout error after about 60 seconds.
API call that fails:
GET /prweb/api/processmining/v1/exports?startDate=2024-11-01&endDate=2025-02-01
Response: 504 Gateway Timeout after 62 seconds
We need this historical data for quarterly trend analysis and compliance reporting. The dataset is approximately 450,000 process instances. Is there a way to implement API pagination for large exports or configure asynchronous export options so we don’t hit gateway timeouts?
For large exports, asynchronous processing is really the best approach. Instead of waiting for the export to complete synchronously, the API should accept the request, generate the export in the background, and notify you when it’s ready. This requires modifying the export service to use Pega’s job scheduler. The API would return a job ID immediately, then you poll a status endpoint until the export completes and download the file.
I tried pagination with limit=10000 but the API doesn’t seem to recognize those parameters - still returns the full dataset (and times out). Are these parameters documented for Pega 8.5 process mining API? Maybe I need to enable pagination in the service configuration?
Pagination is definitely the way to go. The process mining API should support limit and offset parameters. Try adding ?limit=10000&offset=0 to your request, then increment offset for subsequent calls. This retrieves 10,000 records at a time. You’ll need to make multiple API calls and concatenate the results client-side, but each individual call will complete within the timeout window.
Pega 8.5’s process mining export API might not have built-in pagination support depending on your version. Check your API documentation or try using cursor-based pagination with ?pageSize=10000&cursor=next. Alternatively, you could implement date-based chunking - break your 3-month range into weekly exports and merge them. Not elegant, but it works around the timeout limitation.
504 errors indicate the gateway timeout, not Pega itself. Your load balancer or API gateway is probably configured with a 60-second timeout. Large dataset exports can take several minutes to generate. You have two options: increase the gateway timeout configuration (not recommended for APIs) or implement pagination to retrieve data in smaller chunks.