One of the tools I developed and manage for work is populated with data from Cognos on a daily basis. In order to not have to download and upload a file like some sort of IDIOT, I figured it would be good to write a Selenium script to do this for me.
For those of you who and are reading this post because you are my friend from the real world or bike racing or the internet, you might want to stop reading it because it will probably not pertain to you and I’m really writing it so I can remember what the hell I did later on.
PUTTING THE SOLUTION AT THE TOP
If you’re using Selenium in Python, and using the Firefox driver AND you are constantly getting the error message:
Can't load the profile. Profile Dir: [some directory] If you specified a log_file in the FirefoxBinary Constructor, check it for details.
This seemingly fixed my problem: in the file /lib/python2.7/site-packages/selenium/webdriver/firefox/firefox_binary.py, you need to change the timeout used in function _wait_until_connectable to some large static value.
Original function:
def _wait_until_connectable(self, timeout=30):
"""Blocks until the extension is connectable in the firefox."""
count = 0
while not utils.is_connectable(self.profile.port):
if self.process.poll() is not None:
# Browser has exited
raise WebDriverException("The browser appears to have exited "
"before we could connect. If you specified a log_file in "
"the FirefoxBinary constructor, check it for details.")
if count >= timeout:
self.kill()
raise WebDriverException("Can't load the profile. Profile "
"Dir: %s If you specified a log_file in the "
"FirefoxBinary constructor, check it for details."
% (self.profile.path))
count += 1
time.sleep(1)
return True
So, I just added timeout=120 as the first line in the function, overwriting any other value that is passed to it. And now it works!
Source for the solution, thank you raspberrypi.org forum user “glycerine”.
The journey to this point, because why not
So, before the fix, sometimes it worked. Sometimes it didn’t – it was seemingly random. I tried running it with xvfb/pyvirtual display, just xvfb, with an attached display, and probably a bunch of other rabbit holes that people on the internet suggested might work.
@resultsboy it’s fucking working now ¯_(ツ)_/¯ pic.twitter.com/ZYf2EcPSUU
— ryank (@ryantkelly) July 27, 2016
It was really annoying, because then I’d have to download/upload and it was STUPID and ANNOYING and I wanted a ROBOT TO DO IT.
I am running this on a Raspberry Pi 3 on Raspbian because…why not, I guess. This means that I was limited to using Firefox (as the other webdrivers in selenium aren’t available for ARM), so attempting to switch out browsers/webdrivers was not going to work.
I did some more digging, and came across this thread which pointed to this Selenium issue as “oh shit this is the thing that is probably broken.”
So then I tried THAT SHIT, and holy crap things started working! And working nicely with xvfb, so once I’m confident this isn’t consistently being dumb I can take this terrible monitor off my desk and have more space.
I don’t know if it is because I’m running Firefox on a crappy $35 computer, or something to do with the “fun” of Linux, or whatever. But making the driver hang out for a max of two minutes before giving up seems to solve the problem.