I'm running a python script which uses selenium, on an EC2 instance (ubuntu).
Under my AWS plan, I have 2 GB memory. I upgraded to this from the free version after some performance issues with my script. However, when I check the free memory when connected to the ubuntu server, I'm only seeing 348MB of available memory, and 353MB of free memory!
As of now, I'm only running two python scripts, once a day, using crontab. The scripts run through a fairly long array of urls and grabs information from each of them.
base_url = 'https://www.bandsintown.com/en/c/san-francisco-ca?page='
events = []
eventContainerBucket = []
for i in range(1,25):
#cycle through pages in range
driver.get(base_url + str(i))
pageURL = base_url + str(i)
# get events links
event_list = driver.find_elements_by_css_selector('div[class^=_3buUBPWBhUz9KBQqgXm-gf] a[class^=_3UX9sLQPbNUbfbaigy35li]')
# collect href attribute of events in even_list
events.extend(list(event.get_attribute("href") for event in event_list))
allEvents = []
for event in events:
driver.get(event)
//do a bunch of other stuff
driver.quit()
Can anyone see an inherent problem in this code that would be causing memory leak? I would think free memory should go up again when the script has stopped running but it doesn't.
I tried to invoke driver.close()
within the for-loop, so that after the information is retrieved from each URL, the window closes. I was thinking this would help with memory leak - unfortunately, this gave me an error selenium.common.exceptions.InvalidSessionIdException: Message: invalid session id
.
Any help would be greatly appreciated. Am I on the right path with driver.close() or is something else entirely the problem?