Scraping .aspx page

[nilamo]

The structure isn't even that weird, though. Controllers, models, and views have their own subfolders, and urls are rewritten to be pretty. That's the same thing with all decent frameworks in both python and php (...and pretty much any web-based framework).

[Larz60+]

Does Microsoft publish the spec?

[nilamo]

I'm not entirely sure what you're asking, but the framework itself is open source (and so is .net core, which is a more basic version of .net that runs on mac/linux).
https://www.asp.net/
https://github.com/aspnet

[Larz60+]

I'm not entirely sure what you're asking, but the framework itself is open source

Not sure I do either, but this looks like the place to poke around If I want to learn more.
This looks like the kind of thing that you'd need to lock yourself in a padded room for six months
to to absorb.

[nilamo]

It depends on how much of it you want to understand. I look at it a lot like the python std lib... there's a whole lot more there than I'll probably ever need, and just "getting started" is very easy, allowing you to add things in when you actually need them... without understanding any of what's actually happening :p

[Larz60+]

I perhaps don't have the same drive as I did when I was 30, but still like to dig in and understand how things tick.
That is usually a good way to learn, although my girlfriend (now my wife past 40 years) didn't think so when I took her
car apart. But I did get it back together and it ran fine afterwards. My mother didn't appreciate it when I took her
alarm clock apart either.

[owais]

I perhaps don't have the same drive as I did when I was 30, but still like to dig in and understand how things tick. That is usually a good way to learn, although my girlfriend (now my wife past 40 years) didn't think so when I took her car apart. But I did get it back together and it ran fine afterwards. My mother didn't appreciate it when I took her alarm clock apart either.

Did you got your problem solved ?

[Larz60+]

Yes - I'm just finishing up and will post results soon
Snippsat is our resident Guru on web scraping and
gave me the answer.

[Phillisk]

I know this thread is a bit old but I am facing the same issue. I am using scrapy on an ASP site which has pages upto 1000 plus. I need to visit the last page to capture the number of pages available... I then take that and pass it through a loop. I can visit all pages but the problem is that the entire loop carrys the response and viewstate from the last page which doesn't work for pages not visible from the last page on the browser. I need a way to refresh the response after every page but I am stuck.

[Larz60+]

What's the URL?