Conditional redirect? #124

New issue

Open

opened 2025-10-14 18:13:01 -06:00 by navan · 0 comments

navan commented

2025-10-14 18:13:01 -06:00

Owner

Originally created by @FezVrasta on 12/30/2017

Subject of the issue

I'm scraping a page that requires the user to click on a "thanks" button before I can access the info I'm looking for.
If the page has already been "thanked", an "unthank" button will be shown instead. I don't want the scraper to click on the "unthank" obviously.

Right now I'm using a code of this kind:

const x = Xray({
  filters: {
	// this filter makes the "thank/unthank" URL call become always a "thank" call
    // doing so I avoid to unthank anything that was previously thanked
    grateful: url => url && url.replace('withdrawthank', 'thank'),
  },
}).driver(driverWithAuth);

x(
  websiteUrl,
  '.subject > div > span',
  [
    {
      title: 'a',
      url: 'a@href',
      magnets: x(
        'a@href',
        x('.thank_you_button:first-child a@href | grateful', [
          'a@href',
        ])
      ),
    },
  ]
)

The problem with this setup is that I will always have to navigate to an additional page even when the page is already "thanked".

Ideally I'd like to be able to tell x-ray to not navigate to anything but just stay in the same page if the provided URL is maybe undefined or false.

Doing so I could have my filter do url.includes('thank') ? url : false, which will make the scraper hit the thank call only if needed.

Is it possible? If not, could this be added?

*Originally created by @FezVrasta on 12/30/2017* ### Subject of the issue I'm scraping a page that requires the user to click on a "thanks" button before I can access the info I'm looking for. If the page has already been "thanked", an "unthank" button will be shown instead. I don't want the scraper to click on the "unthank" obviously. Right now I'm using a code of this kind: ``` const x = Xray({ filters: { // this filter makes the "thank/unthank" URL call become always a "thank" call // doing so I avoid to unthank anything that was previously thanked grateful: url => url && url.replace('withdrawthank', 'thank'), }, }).driver(driverWithAuth); x( websiteUrl, '.subject > div > span', [ { title: 'a', url: 'a@href', magnets: x( 'a@href', x('.thank_you_button:first-child a@href | grateful', [ 'a@href', ]) ), }, ] ) ``` The problem with this setup is that I will always have to navigate to an additional page even when the page is already "thanked". Ideally I'd like to be able to tell x-ray to not navigate to anything but just stay in the same page if the provided URL is maybe `undefined` or `false`. Doing so I could have my filter do `url.includes('thank') ? url : false`, which will make the scraper hit the thank call only if needed. Is it possible? If not, could this be added?

navan added the

enhancement

labels

2025-10-14 18:13:04 -06:00

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference: github/x-ray#124

No description provided.

Rows
Columns