After facial recognition, I am tackling language issues with Python. Indeed, after image, the other ingredient for a post is mostly text! As a CMS “manufacturer” or PO, I was wondering what advantages I can withdraw from NLP. Concretely, it means exploring and learning Python to improve both user support (FAQs turn as a Chatbot, analyzing User Feedback…) but also think about some editorial features especially with the help of Natural Language Processing (NLP).<\/p>\n
I am wrestling with the subject for too long because there are tons of libraries and tutorials introducing to Python and NLP! Looking for python is an heavy trend. Apparently, Google users in America have searched for Python more often than for Kim Kardashian. So, “creating a chatbot in Python” has become the typical quoted example like “creating a blog” or “hello world” in other language!<\/p>\n
Source: https:\/\/www.economist.com\/science-and-technology\/2018\/07\/19\/python-has-brought-computer-programming-to-a-vast-new-audience<\/a><\/p>\n Like I said at the beginning of this post. The starting idea was simple, how can I aleviate real-world tasks such as: <\/p>\n I found some very contrived examples and some more advanced ones like always. Even though, these examples are oversimplified, there are still caveats for less technical readers, including me, especially when it comes to concepts pertaining to linguistic eg stemming, tokenization, tokenizer, bag of words or Convolutional Neural Network.<\/p>\n I invite to check these very intuitive videos that are good introduction to NLP.<\/p>\n These videos are giving some enlightening on NLP’s concepts such as stemming, tokenization, tokenizer or bag of words or even some explanations on different type of IA\u2019s network such as Convolutional Neural Network and the way to use it.<\/p>\n Here is a posts’digest to start with NLP oriented around 2 basic usages, that can be implemented in a CMS: Practical Use Cases in a CMS’s support and simple techniques to extract Keywords or even “slice” a post.<\/b><\/p>\n Chronologically, the very first library, I explored was the famous NLPs librairie, NTLK. Then I discovered Pytorch, made by Facebook and then Spacy. There will be certainly a more specific article on Spacy because I really like Spacy for its accessibility both in tutorials and in its core values. After all this reading, I selected few articles that were illustrating, at least, some of my personal interests for NLP. This post is a quick overview of that exploration.<\/b><\/p>\n The source code is available on my GitHub account and I am using my own mac plus anaconda to deal and install all the require libraries (https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction<\/a>). All libraries have been installed with the help of Anaconda<\/a>.<\/b><\/p>\n A good usecase for support where you parse a unique user feedback file and retrieve core informations with NLP. This usecase leverage on a bunch of librairies such as Panda, Scipy, Seaborn, scikit-learn and for sure NLTK. It parses a huge document in .tsv format (Tab-separated values).<\/p>\n Source: https:\/\/www.andyfitzgeraldconsulting.com\/writing\/keyword-extraction-nlp\/<\/a><\/p>\n Source: https:\/\/github.com\/andybywire\/nlp-text-analysis<\/a> <\/p>\n My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_1_keyword_extraction_nlp<\/a><\/p>\n Requirement to run the scripts<\/b><\/p>\n Source: https:\/\/www.andyfitzgeraldconsulting.com\/writing\/keyword-extraction-nlp\/<\/a><\/p>\n For me, only first part was interesting, it shows how to use RAKE which stands for Rapid Automatic Keyword Extraction. RAKE extracts keywords that should describe the main topics expressed in a document. <\/p>\n Source: https:\/\/www.airpair.com\/nlp\/keyword-extraction-tutorial<\/a><\/p>\n My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_2_keyword_extraction_nlp_rake<\/a><\/p>\n This article from Ng Wai Foong and some other examples from the great official spaCy documentation show how to quickly get to grip with Spacy.<\/p>\n The script extracting keywords with Spacy is straightforward like the other article from this guy Ng Wai Foong.<\/p>\n Source: https:\/\/medium.com\/better-programming\/extract-keywords-using-spacy-in-python-4a8415478fbf<\/a><\/p>\n My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_3_keyword_extraction_nlp_spacy<\/a><\/p>\n Some miscellaneous linguistic scripts using spaCy. There is much more on their github account and the documantion is terrific.<\/p>\n Source: https:\/\/github.com\/explosion\/spaCy<\/a><\/p>\n My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_4_miscellaneous_examples_nlp_spacy<\/a><\/p>\n A simple attemps with a librairie newspaper. The Python librairie gives the ablility to slice up any post online. In the script, as an example, I am using one of my blog’s post.<\/p>\n Source: https:\/\/newspaper.readthedocs.io\/en\/latest\/<\/a><\/p>\n My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_5_playing_with_newspaper_post_scraping_curation<\/a><\/p>\n If you need to check the import of newspaper<\/b><\/p>\n We left the Keyword Extraction for ChatBot. Turning my FAQ to a ChatBot with the help of Pytorch and NLTK. It is a very intuitive tutorial and the videos are making the rest. Source: https:\/\/www.python-engineer.com\/videos\/chatbot-pytorch\/<\/a><\/p>\n My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_6_chatbot_with_pytorch<\/a><\/p>\n A different ChatBot Build on TensorFlow from a .txt file.<\/p>\n Source: https:\/\/medium.com\/x8-the-ai-community\/build-your-first-chatbot-in-python-334247814900<\/a><\/p>\n My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_7_chatbot_with_tensorflow<\/a><\/p>\n I found a more advanced Chatbot tutorial with Pytorch. Be careful with the n_iteration value because it requires a lot of space disk! I was forced to downsize the training but then the chatbot sucks a little bit. Anyway, the example is great. <\/p>\n Check My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_8_chatbot_tutorial_pytorch<\/a><\/p>\n A second article from Ng Wai Foong. It is about Text Summarization with TF-IDF (Term Frequency-Inverse Data Frequency). It leverages on Spacy and the result is immediate<\/p>\n Source: https:\/\/medium.com\/better-programming\/extractive-text-summarization-using-spacy-in-python-88ab96d1fd97<\/a><\/p>\n My files: https:\/\/github.com\/bflaven\/BlogArticlesExamples\/tree\/master\/python_nlp_explorations_chatbot_keywords_extraction\/article_9_text_summarization_using_spacy<\/a><\/p>\n Let’s step back a little bit to think a minute about IA consequences. These IA tools exerts an undeniable fascination. Why? These are new tools that actually begin to think and act on its own. The idea that these tools will make decisions and undertake actions on their own is fascinating and scary at the same time. I was wondering if any critical thinking existed towards the deafening consensus on AI? Indeed, IA can been seen as the ultimate market achievement where, reduced as consumers, we only take decisions with utilitarian goals, “obeying” to IA.<\/p>\n Regarding NLP, the disturbing thing is the familiar form that this injunction takes. The Chatbot speaks to you, the NLP writes and advises you with your own words… This is step forward to a very persuasive soft power. So, is the very idea of rebelling still even exist as it sounds ludicrous, to fight with a friend!<\/p>\n What I mostly remember from this reading: Even though it is nowadays almost impossible to avoid IA, that’s always good to read opinions against the mainstream way of thinking so you\u2019ll be aware of the potential IA\u2019s threats!<\/p>\n I wonder more and more if the target has not become to even drop PHP to build a web application. Is refactoring a PHP legacy Code in Python is an option?<\/p>\n Indeed, building a web application, I do not even talk about a website seems to be easy nowadays. You can gather an effective SPA (Single Page Application) in a very short time but providing meaningful and advanced features for a CMS is much trickier! <\/p>\n To be totally transparent, a simple question is spinning around in my head: How can I add some “intelligent” functionalities, using these Python libraries, to an existing CMS made in PHP (Laravel or Symfony for instance)? Apparently, the way seems to build a separate API in Python that will brigde with PHP!<\/p>\n Like I said in my previous post, these NLP libraries are not only enabling new tasks to be made but these libraries can even carry out tasks like a real human such as me a P.O for a Backoffice! Great, I am outsourcing myself.<\/p>\n A reminder for useful commands with Anaconda<\/p>\n This reading gave me an overview of the Python’s possibilities in terms of text understanding. NLP seems to be progressing more and more every day and remains fairly accessible if you do not pretend to be a specialist! It would be a shame to do without it while waiting to see progress in text’s generation with BERT for example.<\/p>\n To remove doubts about the NLP’s potential, this post underwent both a keyword extraction and summarization operations with the help of 2 scripts given here as example. The result is not bad! It is a true crutch for a rookie journalist like me! You can see below the result.<\/b><\/p>\n API python + vue.js<\/b><\/p>\n Python goes on web…<\/b><\/p>\n If you need to go to the web<\/p>\n Some extra ressources<\/b><\/p>\n After facial recognition, I am tackling language issues with Python. Indeed, after image, the other ingredient for a post is mostly text! As a CMS… <\/p>\n\n
\n
https:\/\/www.youtube.com\/watch?v=sqDHBH9IjRU<\/a><\/li>\n
ttps:\/\/github.com\/python-engineer\/pytorchTutorial<\/a><\/li>\n
https:\/\/course.spacy.io\/en\/<\/a><\/li>\n<\/ul>\n1. Keyword Extraction<\/h2>\n
A beginner’s guide to keyword extraction with natural language processing (article_1_keyword_extraction_nlp)<\/h3>\n
\r\n\r\n# look in in https:\/\/anaconda.org\/conda-forge\/ to find the command.\r\n\r\n# pandas for data analysis and manipulation \r\nconda install -c anaconda pandas\r\n\r\n# nltk, statistical natural language processing \r\nconda install -c anaconda nltk\r\n\r\n\r\n# matplotlib is a python 2D plotting library\r\nconda install -c anaconda matplotlib\r\n\r\n\r\n# seaborn visualization library based on matplotlib\r\nconda install -c anaconda seaborn\r\n\r\n\r\n# pillow imaging Library, image processing capabilities to your Python interpreter.\r\nconda install -c anaconda pillow\r\n\r\n\r\n\r\n# an imaging library to create word cloud visualizations\r\nconda install -c conda-forge wordcloud\r\n\r\n\r\n# install sklearn or scikit-learn\r\nconda install -c anaconda scikit-learn\r\n\r\n<\/pre>\n
NLP keyword extraction tutorial with RAKE and Maui (article_2_keyword_extraction_nlp_rake)<\/h3>\n
Extract Keywords Using spaCy in Python (article_3_keyword_extraction_nlp_spacy)<\/h3>\n
Miscellaneous examples with spaCy (article_4_miscellaneous_examples_nlp_spacy)<\/h3>\n
Scraping Post<\/h2>\n
Newspaper: Article scraping & curation (article_5_playing_with_newspaper_post_scraping_curation)<\/h3>\n
\r\n$ python\r\n<\/pre>\n
\r\n>>> import newspaper\r\n>>> newspaper.__version__\r\n>>> exit()\r\n<\/pre>\n
ChatBot<\/h2>\n
ChatBot With PyTorch – NLP And Deep Learning (article_6_chatbot_with_pytorch)<\/h3>\n
\nCertainly, I was not rapt in ecstasy by the chatbot ability but there is a lot of promises for Chatbots, supposed to be handling fairly complex conversation with humans and so using a lot of Natural Language Processing techniques in order to understand the human’s requests.<\/p>\nBuild Your First Chatbot in Python (article_7_chatbot_with_tensorflow)<\/h3>\n
Chatbot tutorial by Matthew Inkawhich (article_8_chatbot_tutorial_pytorch)<\/h3>\n
\nhttps:\/\/pytorch.org\/tutorials\/beginner\/chatbot_tutorial.html<\/a><\/p>\nText Summarization Using spaCy in Python (article_9_text_summarization_using_spacy)<\/h3>\n
Some critics about IA<\/h2>\n
\nI found some opponents, on a philosophical point of view. Even though, IA fanatics report that the AI promise is to “Humanize the machine, not mechanize the User”, the main critic remains the AI’s “injunctive power”. Combined with consent, it makes an unstoppable combination to turn us mankind into passive and obedient sheeps! By the way, the GAFAS, that promote IA, never really assumed they were the bad guy.<\/p>\n
\n– IA is a threat to humanity, especially our free will.
\n– IA is the ultimate version of “Invisible hand”, so criticizing IA seems to be the way to “burn down” the system aka capitalism, GAFAS (Facebook, Google, Amazon… etc. that are mostly behind the IA libraries and expect something in return: your datas so you can be profiled.<\/p>\n
https:\/\/www.amazon.fr\/Lintelligence-artificielle-lenjeu-si%C3%A8cle-antihumanisme\/dp\/2373090503<\/a><\/li>\n
https:\/\/onezero.medium.com\/peter-w-singer-explains-why-his-robot-revolution-is-inevitable-2b858a68151f<\/a><\/li>\nWhat’s next? How can I use NLP?<\/h2>\n
Using Anaconda<\/h2>\n
\r\n# Check Anaconda installation\r\nconda --version\r\n\r\n# Update conda\r\nconda update -n base -c defaults conda\r\n\r\n# Create an environment\r\n# environment with Python 3.5\r\nconda create --name myEnvironmentOne python=3.5\r\n\r\n# launch a script named 006_nltk_cookbook_test.py\r\n--- python 006_nltk_cookbook_test.py\r\n\r\n# Get into the env named myEnvironmentOne\r\nconda activate myEnvironmentOne\r\n\r\n# get out from an env\r\nconda deactivate\r\n\r\n# By default you are in the base env no need to activate\r\n# conda activate base\r\n<\/pre>\n
In conclusion<\/h2>\n
\r\n# result for post tags\r\npython, nlp, spacy, ia, chatbot\r\n\r\n# summarization result\r\nThese videos are giving some enlightening on nlp's concepts such as stemming, tokenization, tokenizer or bag of words or even some explanations on different type of ia\u2019s network such as convolutional neural network and the way to use it.here is a posts'digest to start with nlp oriented around 2 basic usages, that can be implemented in a cms: practical use cases in a cms's support and simple techniques to extract keywords or even \"slice\" a post. Rake extracts keywords that should describe the main topics expressed in a document. An article from ng wai foong and some other examples from the great official spacy documentation show how to quickly get to grip with spacy. Concretely, it means exploring and learning python to improve both user support (faqs turn as a chatbot, analyzing user feedback...) but also think about some editorial features especially with the help of natural language processing (nlp).\r\n<\/pre>\n
Read more<\/h2>\n
\n
https:\/\/medium.com\/predict\/create-your-chatbot-using-python-nltk-761cd0aeaed3<\/a><\/li>\n
https:\/\/nlpforhackers.io\/language-models\/<\/a><\/li>\n
https:\/\/heartbeat.fritz.ai\/deploying-a-text-classification-model-using-flask-and-vue-js-25b9aa7ff048<\/a><\/li>\n
https:\/\/stackabuse.com\/text-generation-with-python-and-tensorflow-keras\/<\/a><\/li>\n
https:\/\/pytorch.org\/tutorials\/beginner\/text_sentiment_ngrams_tutorial.html<\/a><\/li>\n
https:\/\/monkeylearn.com\/text-analysis\/<\/a><\/li>\n
https:\/\/mccormickml.com\/2019\/05\/14\/BERT-word-embeddings-tutorial\/<\/a><\/li>\n
https:\/\/gist.github.com\/brianspiering\/64b2256f25880a97936c198955b437e1<\/a><\/li>\n
\n
https:\/\/hackernoon.com\/list-of-free-resources-to-learn-natural-language-processing-5bc4b76db552<\/a><\/li>\n
https:\/\/blog.usejournal.com\/how-i-built-and-launched-an-ai-product-for-under-100-6284646ec56b<\/a><\/li>\n
https:\/\/nlpforhackers.io\/chatbots-introduction\/#more-8595<\/a><\/li>\n
https:\/\/testdriven.io\/blog\/developing-a-single-page-app-with-flask-and-vuejs\/<\/a><\/li>\n
https:\/\/www.smashingmagazine.com\/2019\/02\/interactive-weather-dashboard-api-vue-js\/<\/a><\/li>\n
https:\/\/github.com\/e0ne\/docker-vuejs-python-nginx<\/a><\/li>\n
https:\/\/scotch.io\/bar-talk\/
\nbuilding-a-google-search-clone-spa-with-vue-and-flask<\/a><\/li>\n
https:\/\/towardsdatascience.com\/training-tensorflow-object-detection-api-with-custom-dataset-for-working-in-javascript-and-vue-js-6634e0f33e03<\/a><\/li>\n
https:\/\/flask-restful.readthedocs.io\/en\/latest\/<\/a><\/li>\n
https:\/\/github.com\/huggingface\/transformers<\/a><\/li>\n
https:\/\/www.curiousily.com\/posts\/sentiment-analysis-with-bert-and-hugging-face-using-pytorch-and-python\/s<\/a><\/li>\n
https:\/\/flask.palletsprojects.com\/en\/1.1.x\/<\/a><\/li>\n
https:\/\/www.djangoproject.com\/<\/a><\/li>\n
https:\/\/github.com\/amueller\/word_cloud<\/a><\/li>\n
https:\/\/www.analyticsvidhya.com\/blog\/2019\/07\/openai-gpt2-text-generator-python\/<\/a><\/li>\n
https:\/\/www.analyticsvidhya.com\/blog\/2020\/07\/transfer-learning-for-nlp-fine-tuning-bert-for-text-classification\/<\/a><\/li>\n
https:\/\/towardsdatascience.com\/how-to-train-your-chatbot-with-simple-transformers-da25160859f4<\/a><\/li>\n
https:\/\/www.youtube.com\/c\/PythonEngineer\/videos<\/a><\/li>\n
https:\/\/github.com\/python-engineer<\/a><\/li>\n
https:\/\/github.com\/topics\/text-mining<\/a><\/li>\n
https:\/\/github.com\/JulienHeiduk\/bert-generative-text<\/a><\/li>\n
https:\/\/github.com\/vi3k6i5\/flashtext<\/a><\/li>\n
https:\/\/pythonprogramming.net\/basic-flask-website-tutorial\/?completed=\/practical-flask-introduction\/
\n<\/a><\/li>\n
https:\/\/github.com\/jalajthanaki\/NLPython<\/a><\/li>\n
https:\/\/www.thepythoncode.com\/article\/text-generation-keras-python<\/a><\/li>\n
https:\/\/github.com\/x4nth055\/pythoncode-tutorials\/tree\/master\/machine-learning\/nlp\/text-generator<\/a><\/li>\n
https:\/\/wordlift.io\/blog\/en\/keyword-suggestion-tool-tensorflow\/<\/a><\/li>\n
https:\/\/seobutler.com\/how-to-automate-keyword-research-with-apis-python-scripts\/
\n<\/a><\/li>\n
https:\/\/github.com\/csurfer\/rake-nltk<\/a><\/li>\n
https:\/\/github.com\/JRC1995\/TextRank-Keyword-Extraction<\/a><\/li>\n
https:\/\/github.com\/chiphuyen\/lazynlp<\/a><\/li>\n
https:\/\/www.hallada.net\/2017\/07\/11\/generating-random-poems-with-python.html<\/a><\/li>\n
https:\/\/www.pythonprogramming.in\/<\/a><\/li>\n
https:\/\/medium.com\/@datamonsters\/text-preprocessing-in-python-steps-tools-and-examples-bf025f872908<\/a><\/li>\n
https:\/\/medium.com\/towards-artificial-intelligence\/text-mining-in-python-steps-and-examples-78b3f8fd913b<\/a><\/li>\n
https:\/\/www.youtube.com\/watch?v=w36-U-ccajM<\/a><\/li>\n
\n
https:\/\/www.youtube.com\/watch?v=imPpT2Qo2sk<\/a><\/li>\n
https:\/\/github.com\/Sentdex<\/a><\/li>\n
https:\/\/kavita-ganesan.com\/<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"