Paginate and limit based on number of pages #93

New issue

Closed

opened 2025-10-14 18:09:12 -06:00 by navan · 0 comments

navan commented

2025-10-14 18:09:12 -06:00

Owner

Originally created by @Globerada on 4/19/2018

Hi.
I have not found this information in the docs.

How can I achieve a paginate based on the numbers of pages that the URL have?
Below is the example that I am using. Instead of a high limit so I can crawl all the pages, how can I put a valid limit base on the real number of pages?

x('http://www.example.com/products', 'div.products_details_container', data)
.paginate('.pagination a:last-Child@href')
.limit(999)
.write('results.json');

*Originally created by @Globerada on 4/19/2018* Hi. I have not found this information in the docs. How can I achieve a paginate based on the numbers of pages that the URL have? Below is the example that I am using. Instead of a high limit so I can crawl all the pages, how can I put a valid limit base on the real number of pages? x('http://www.example.com/products', 'div.products_details_container', data) .paginate('.pagination a:last-Child@href') .limit(999) .write('results.json');

navan

2025-10-14 18:09:13 -06:00

closed this issue
added the
tech support

tech support

tech support

tech support
labels

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: github/x-ray#93

No description provided.

Rows
Columns