Repository navigation
HtmlUnit 4.7.0 Failing To Load Forms #932
Description
If you inspect https://www.asr.pima.gov/ using Chrome on Windows 11, you will see that the page has two forms.
But if you run the following code on Apache Netbeans 25 using JDK 24 and HtmlUnit 4.7.0, you will see that no forms are loaded:
package pima;
import org.htmlunit.*;
import org.htmlunit.html.*;
public class Pima {
public static void main(String[] args) {
try (final WebClient webClient = new WebClient(BrowserVersion.FIREFOX)) {
java.util.logging.Logger.getLogger("org.htmlunit").setLevel(java.util.logging.Level.OFF);
webClient.getOptions().setCssEnabled(false);
webClient.getOptions().setThrowExceptionOnScriptError(false);
webClient.getOptions().setThrowExceptionOnFailingStatusCode(false);
HtmlPage page = webClient.getPage("https://www.asr.pima.gov/");
webClient.waitForBackgroundJavaScriptStartingBefore(1_000_000);
page = (HtmlPage) page.getEnclosingWindow().getEnclosedPage();
System.out.println("page.asXml() = " + page.asXml());
}
catch (Exception e) {
System.out.println("Exception: " + e.toString());
}
}
}
The following does NOT fix the issue:
Adding the following line before "HtmlPage page ...":
webClient.setAjaxController(new NicelyResynchronizingAjaxController());
Activity
rbri commentedon Feb 21, 2025
I'm currently working on deploying the 4.10 release. Will have a look after that.
AbdenourAchab commentedon Feb 21, 2025
By the way, the reason I am using HtmlUnit 4.7.0 instead of 4.9.0 is that, when I downloaded 4.9.0, I was unable to find the numerous jar files like 4.7.0 had. I found only one jar far, and, when I added that single file to my project library in Netbeans, I was getting errors such as package htmlunit not found. Hopefully, the upcoming 4.10.0 release will have all the jar files I need to upgrade from 4.7.0 to 4.10.0
rbri commentedon Feb 21, 2025
@AbdenourAchab which file have you downloaded from where,?
AbdenourAchab commentedon Feb 22, 2025
@rbri , I downloaded htmlunit-4.9.0-bin.zip, htmlunit-4.9.0-src.zip and htmlunit-4.9.0-OSGi.jar from https://sourceforge.net/projects/htmlunit/. I just double checked, and htmlunit-4.9.0-bin\lib does have all the required jar files. For whatever reason, when I copied the extracted htmlunit-4.9.0-bin folder from my Downloads folder to my libraries\external\htmlunit\4.9.0 folder, my Windows only copied the apidocs subfolder and the LICENSE file. It didn't copy the lib subfolder. Maybe I tried to do the copying before the extraction of the zip file was completed.
I have just upgraded my HtmlUnit from 4.7.0 to 4.9.0, and the problem described in the present ticket persists.
rbri commentedon Feb 22, 2025
Thanks for checking
AbdenourAchab commentedon Feb 25, 2025
@rbri I have upgraded to HtmlUnit 4.10.0, and the problem described in the present ticket still persists.
rbri commentedon Feb 26, 2025
Hi @AbdenourAchab,
when facing problems, disabling the loggger is the wrong approach.....
Have done some checks and this seems to be a SPA based on angular. This means this uses huge js libraries and browser features not really supported by HtmlUnit so far. We are constantly working on that but at the moment there is no real hope for a solution during the next weeks.
Sorry for that....
AbdenourAchab commentedon Feb 26, 2025
Hi @rbri ,
I am not technical enough to understand what the logger would tell me. And I don't need a fix in the next few weeks. I only need a fix 11 months from now. The reason is, from February 1 until about a week ago, I have extracted enough information from the website using HtmlUnit for my 2025 needs.
Then, about a week ago, the website went offline for about half a day. And, when it came back online, I was no longer able to get my HtmlUnit code to work on it.
So, it seems like the website developer introduced a week ago some new website development technology that HtmlUnit is not yet able to handle.
Abdenour