{"id":11616,"date":"2021-01-03T12:47:08","date_gmt":"2021-01-03T11:47:08","guid":{"rendered":"http:\/\/flaven.fr\/?p=11616"},"modified":"2026-09-16T10:11:59","modified_gmt":"2026-09-16T08:11:59","slug":"web-scraping-beautifulsoup-selenium-various-explorations-in-web-scraping-with-python-and-jumping-timidly-in-surveillance-capitalism","status":"publish","type":"post","link":"https:\/\/flaven.fr\/2021\/01\/web-scraping-beautifulsoup-selenium-various-explorations-in-web-scraping-with-python-and-jumping-timidly-in-surveillance-capitalism\/","title":{"rendered":"Web scraping, Beautifulsoup, Selenium – Various explorations in web scraping with Python and jumping timidly in surveillance capitalism"},"content":{"rendered":"

I was in a hurry, to unleash the last post of the year’s 2020. Finally, I postponed the publication as most of the issues addressed in this post are so 2021!
\nTo make it short, last year, I was asked by a friend to apply web scrapping in a competitors\u2019 spying campaign. I should say Business Intelligence, it is more corporate and politically correct! Anyway, it does not surprise me so much that learning Testing, Automation and Python will lead me to this area partially included in Surveillance Capitalism Phenomenon. After all, this is still information. Precisely, in my case, the guy wanted solely a powerful automation that will collect, process and order a lot of public information available on webpages. [SIC]<\/p>\n

I won’t tell you, how, what and why but let me just say that with the help of Python, NLP, Web scraping… I can ensure you that we pinned down the crook.<\/p>\n

In conclusion, transparency dictatorship that prevails in surveillance capitalism has its best days ahead of it. Indeed, Digital has become a Personal Data Motherload!<\/p>\n

On a more personal POV \ud83d\ude42 Here is what I learnt from this experience: <\/p>\n

    \n
  1. It has become tremendously easy to “hijack” or divert technologies from their first uses to make a Surveillance Swiss Army Knife even for an absolute beginner like me.<\/li>\n
  2. Learning is rewarding but practising is even better. With this usecase, I was suddenly no longer facing theory but a complex reality. I took it as an opportunity to test my solving problem capability. That was a true exercise to practice what I have just learned (Python mostly).<\/li>\n<\/ol>\n

    Hope that this quick and mundane introduction did not have a chilling effect on you! Here is some commands and scripts about Web scraping, all in Python. I have purposely excluded the things made with NLP or using JavaScript End to End Testing Framework such as CodeceptJS or Cypress that are beyond the scope of this post.<\/b><\/p>\n

    You can get all code and files on my github account in webscraping_with_python<\/a><\/b><\/p>\n

    By the way, a great source of inspiration, all the works of Al Sweigart @https:\/\/inventwithpython.com\/<\/a>. “Automate the Boring Stuff with Python” has been my recent bedside reading!<\/b><\/p>\n

    Some elements about Web Scraping<\/h3>\n

    Here are various examples that automate the browser usage in order to get a map on Google Map or perform a search on Google, or download and save file. I made some attempts to grab HTML content with the help of BeautifulSoup, so all filenames with the pattern: slurpThatSoup_1.py, slurpThatSoup_2.py… are handling a BeautifulSoup Object from HTML<\/p>\n

    FYI, BeautifulSoup (bs4) is one of the best Web Scraping library in Python is BeautifulSoup. You can find more on the official websites and documentation.<\/p>\n

      \n
    1. Beautiful Soup: We called him Tortoise because he taught us.https:\/\/www.crummy.com\/software\/BeautifulSoup\/<\/a><\/li>\n
    2. Beautiful Soup Documentation: https:\/\/www.crummy.com\/software\/BeautifulSoup\/bs4\/doc\/<\/a><\/li>\n<\/ol>\n

      As a huge supporter of testing, I made an extended usage of selenium Module to control the Browser, leveraging on Selenium\u2019s WebDriver Methods to Finding HTML Elements.
      \nEven so, it is sometime easier to divert testing frameworks such as CodeceptJS to perform actions in browser instead of using Selenium.<\/b><\/p>\n

      Due to space disk limitations, I am using Anaconda to build my personal environment on an external disk. See the following previous post about conda install at Python, Anaconda, WordPress, JSON, JSON-SCHEMA \u2013 Messy post with few practices and feedback from my P.O experience.<\/a><\/b><\/p>\n

      Commands to install the required python libraries<\/b><\/p>\n

      \r\n\r\n# install requests and beautifulsoup4\r\npip install requests\r\npip install beautifulsoup4\r\n\r\n# install pyperclip\r\npip install pyperclip\r\n\r\n# check if pyperclip is working\r\n# go to a python console\r\n>>> import pyperclip\r\n>>> import requests\r\n>>> import beautifulsoup4\r\n\r\n<\/pre>\n
      Check if the installed librairies are working<\/b><\/p>\n
      \r\n# check if pyperclip is working\r\n# go to a python console\r\n>>> import pyperclip\r\n>>> import requests\r\n>>> import beautifulsoup4\r\n\r\n# if you get no errors, you are fine...\r\n<\/pre>\n
      Installing the selenium Module<\/b><\/p>\n
      \r\n# required installation\r\npip3 install selenium\r\n\r\n# better using homebrew\r\nbrew install geckodriver\r\n\r\n# check version\r\ngeckodriver --version\r\n\r\n# which \r\nwhich geckodriver\r\n\r\n# Source: https:\/\/www.dev2qa.com\/how-to-resolve-webdriverexception-geckodriver-executable-needs-to-be-in-path\/\r\n\r\n<\/pre>\n
      Conclusion : Web Scraping is just an appetizer, that’s called not see the wood for the trees. It gave me the occasion to deepen a subject that matters to me : AI models generating misinformation or giving illusions of meaning. This time, I was chasing a crook but next time I’ll be the liar or the crook with the help these technologies, who knows? Ethics are melting like the polar ice! See below in Read More section advanced posts on the subject.<\/b><\/p>\n
      Read more<\/h2>\n