One of my favorite things about code is you can automate some of your life, or just speed up the process on many tedious tasks. Here are some tips and ticks as well as my process Ive gone through to help automated many things in my life.
In the beginning.
My very first real attempt at automating was Craigslist. I had just started freelancing and putting free ads on Craigslist but you would really have to renew the ads every few days to push them back to the top since so many ads were published everyday. I tried for a couple years to do this via form submission and cURL, but with no luck. Being determined I spend a whole 2 years trying to figure this out. Enter iMacros.
Chrome Download
Firefox Download
iMacros is a very good place to start for people with no coding skills, or ones that have a little javascript know how like front end developers.
iMarcos lets you use a browser extension to help automate the web browser. This let me sign in, go to particular page and hit the renew button. The simplest way to write an imacro was you just hit record, do the action and it saved it as a macro. Once this was done I set a timer/scheduler on my computer to run this iMacro once a day at 4pm.
One cool thing that iMacros let you do is also use javascript to expand on the functionality and logic of the macro. So I started using it for other tasks mixed with javascript. The main example was using it for dating sites. I was burnt out on sending emails and wading through dating profiles. So I wrote a macro to check the profile for keywords (or negative keywords). Things I wasn’t looking for in a date. Check for particular cities, since the radius feature was too broad. Also checked the length of the profile to make sure they had at least a couple sentences. This meant they were serious about dating. And if they all passed, the marco would send an automated introduction email to that profile. I set this to check 50 profiles a day. The reply emails slowly trickled in. And I felt like I had just solved a part time job! That’s how much time online dating was taking up in my life.
The one down fall of iMacros is was when the script runs it fires up your browser and did all the animation on screen. And this started becoming a problem when I would do screen shares with clients reviewing their projects or giving them training on how to manage their website. Enter phantomjs.
Second Phase with headless
Phantomjs is a headless browser that runs with node.js. It’s all javascript driven which made for an easy transition as my iMacros had already been written with javascript logic. I bought a raspberrypi and installed nodejs, phantomjs, and casperjs. Then reformatted my code for my iMacros and set a crontab (scheduler) to run once a day.
So from then on I had things running remotely, and headless. With this expansion I was able to create some other automated things. I made a twitter bot that auto tweets for me. Also for scheduling tweets from a spreadsheet. And most importantly login and scan emails for followers and then send them a marketing DM for an ebook I wrote.
I made a bot to check crypto prices across multiple exchanges and compare the price difference for the opportunity of arbitrage. Phantomjs was great but I started finding limitations. Mostly when dealing with cookies or dynamic javascript elements on the page. Enter Selenium.
Most flexible option for automation
Selenium uses a web driver and utilized an actual browser for doing it’s tasks. I am still fresh into Selenium, and learning Python which is the combination I have chose to learn. But so far I have been able to get around most of the limitations that I was having with phantomjs. Primary the ability to add cookies.
Here are some tips about selenium. For adding cookies, you load the page, clear the cookies, inject or add your cookies, then reload the page. (there might be a better way but this is how I got it working)
# Login
url = 'https://domain.com'
driver.get(url)
driver.delete_all_cookies()
# Adds the cookie into current browser context
driver.add_cookie({"name":"name_of_cookie", "value":"value_of_cookie"})
driver.get(url)
Go undetected
The other thing I ran into was a couple sites were blocking my bot by detecting it was a headless/automated browser. These are what fixed it for me. First you need to “unset” some options in Chrome driver. As well as give a unique user agent.
from selenium import webdriver # for webdriver
from selenium.webdriver.chrome.options import Options # for suppressing the browser
options = webdriver.ChromeOptions()
# Set headless and alter user agent
options.add_argument('headless')
options.add_argument("window-size=1280,800")
options.add_argument('user-agent=Mozilla/5.0 (Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/74.0.3729.169 Safari/537.36')
# #Removes navigator.webdriver flag
options.add_experimental_option("excludeSwitches", ["enable-automation"])
options.add_experimental_option('useAutomationExtension', False)
#For ChromeDriver version 79.0.3945.16 or over
options.add_argument("--disable-blink-features=AutomationControlled")
driver = webdriver.Chrome(options=options, executable_path="/pathto/altered/chromedriver")
Lastly you actually need to alter the chromedriver itself. Which is why I have this line.
driver = webdriver.Chrome(options=options, executable_path="/pathto/altered/chromedriver")
First locate and copy your chromedriver. Mine was located:
/usr/loca/bin/chromedriver
Then open the copy version and you need to do a find and replace on cdc_ as long as you use 3 letters to replace it, then it won’t break the binary. By using VIM I opened the new chromedriver and type
:%s/cdc_/xxx_/g
Then just need to save with
:wq!
Then just make sure you set executable_path to the edited version and you should go undetected. Of course it’s always a good idea to put some random sleeps, or built in selenium waits to make it look more human. But this should get you past most bot detection.