{"id":10794,"date":"2016-12-30T12:24:56","date_gmt":"2016-12-30T11:24:56","guid":{"rendered":"http:\/\/flaven.fr\/?p=10794"},"modified":"2017-10-17T22:35:05","modified_gmt":"2017-10-17T20:35:05","slug":"elk-elasticsearch-logstash-kibana-exemples-de-data-analysis-a-laide-de-elk","status":"publish","type":"post","link":"https:\/\/flaven.fr\/2016\/12\/elk-elasticsearch-logstash-kibana-exemples-de-data-analysis-a-laide-de-elk\/","title":{"rendered":"ELK, elasticsearch, logstash, kibana &#8211; Exemples de data analysis \u00e0 l&#8217;aide de ELK"},"content":{"rendered":"<p>Il existe un trio remarquable pour la lecture des fichiers de logs. Triptyque qui rend intuitif et parlant un certain nombre d&#8217;informations-cl\u00e9s notamment si vous supervisez des serveurs. Ce trio se nomme elasticsearch, logstash, kibana et est plus connu sous l&#8217;acronyme de ELK&nbsp;*. <\/p>\n<p><i>*&nbsp;ELK qui d\u00e9signe soit dit en passant aussi un cerf en anglais d&#8217;o\u00f9 l&#8217;image.<\/i><\/p>\n<p>Toutefois, si vous n&#8217;avez pas de serveurs \u00e0 superviser, ce qui est le cas de la plupart d&#8217;entre nous, cet outil peut servir \u00e0 tout \u00e0 fait autre chose. La preuve nous en donn\u00e9 par le blog d&#8217;elasticsearch avec des \u00e9tudes de cas assez \u00e9tonnantes sur l&#8217;utilisation de cet ensemble ELK pour l&#8217;analyse de donn\u00e9es (data analysis).<br \/>\nC&#8217;est en substance ce que propose deux des d\u00e9monstrations suivantes :<\/p>\n<ol>\n<li>Using Elastic Stack to Analyze NYPD Motor Vehicle Collision Data<br \/><a href=\"https:\/\/www.elastic.co\/blog\/byodemos-new-york-city-traffic-incidents\" target=\"_blank\">https:\/\/www.elastic.co\/blog\/byodemos-new-york-city-traffic-incidents<\/a><\/li>\n<li>Kibana 4 &amp; Civic Hacking: Investigating Campaign Contributions<br \/><a href=\"https:\/\/www.elastic.co\/blog\/kibana-4-and-civic-hacking-investigating-campaign-contributions\" target=\"_blank\">https:\/\/www.elastic.co\/blog\/kibana-4-and-civic-hacking-investigating-campaign-contributions<\/a><\/li>\n<\/ol>\n<p>Pour chacune de ces d\u00e9monstrations, il existe un d\u00e9p\u00f4t git qui fournit la totalit\u00e9 du code.<\/p>\n<ul>\n<li>Using Elastic Stack to Analyze NYPD Motor Vehicle Collision Data : <a href=\"https:\/\/github.com\/elastic\/examples\/tree\/master\/ElasticStack_nyc_traffic_accidents\" target=\"_blank\">https:\/\/github.com\/elastic\/examples\/tree\/master\/ElasticStack_nyc_traffic_accidents<\/a><\/li>\n<li>Kibana 4 &amp; Civic Hacking: Investigating Campaign Contributions : <a href=\"https:\/\/github.com\/elastic\/examples\/tree\/master\/ElasticStack_usfec\" target=\"_blank\">https:\/\/github.com\/elastic\/examples\/tree\/master\/ElasticStack_usfec<\/a>\n<\/li>\n<\/ul>\n<p><b>L&#8217;installation de l&#8217;ensemble est un peu fastidieux mais cela peut fournir des informations et des rapprochements \u00e9tonnants pour traiter une tr\u00e8s grand nombre d&#8217;information. C&#8217;est un outil vraiment puissant dans ce que l&#8217;on nomme d\u00e9sormais le data-journalisme, qui s&#8217;apparente \u00e0 tout ce qui est relatif \u00e0 la data intelligence comme le vante Cambridge Analytica, grand ordonnateur de la victoire de Donald Trump !<\/b><\/p>\n<p>Le plus court chemin pour installer kibana, elasticsearch et consorts, c&#8217;est de passer par homebrew.<\/p>\n<p>Il est n\u00e9cessaire d&#8217;avoir des instances d&#8217;elasticsearch et de kibana en fonctionnement.<\/p>\n<h4>Installer elasticsearch<\/h4>\n<p>Voil\u00e0 dans  le d\u00e9sordre quelques commandes utiles pour l&#8217;installation d&#8217; elasticsearch.<\/p>\n<pre lang=\"bash\">\r\n# installation d'elasticsearch \u00e0 l'aide de homebrew\r\nbrew -v\r\nbrew doctor\r\nbrew update\r\nbrew install elasticsearch\t\r\n<\/pre>\n<p><b>Pour lancer ensuite elasticsearch dans un onglet de la console<\/b><\/p>\n<pre>elasticsearch<\/pre>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/flaven.fr\/wp-content\/uploads\/2016\/12\/elasticsearch_1.jpg\" alt=\"ELK, elasticsearch, logstash, kibana - Exemples de data analysis \u00e0 l'aide de ELK\" width=\"500\" height=\"313\" \/><\/p>\n<h4>Installer kibana<\/h4>\n<p>L\u2019installation de kibana est indiqu\u00e9 sur la page de d\u00e9marrage, le plus simple est de t\u00e9l\u00e9charger puis de d\u00e9compresser l\u2019archive kibana<\/p>\n<pre lang=\"bash\">\r\n# installation de kibana\r\nshasum kibana-5.1.1-darwin-x86_64.tar.gz \r\ntar -xzf kibana-5.1.1-darwin-x86_64.tar.gz\r\n<\/pre>\n<p><b>D\u00e9marrer kibana depuis un r\u00e9pertoire nomm\u00e9 kibana_dashboard<\/b><\/p>\n<pre lang=\"bash\">\r\ncd [chemin-d-installation-vers-le-r\u00e9pertoire]\/kibana_dashboard\/\r\ncd kibana\/ \r\n<\/pre>\n<p>Une fois, lancer elasticsearch dans un onglet de la console, il vous faudra aussi lancer kibana dans un autre onglet de la console.<\/p>\n<pre lang=\"bash\">\r\n.\/bin\/kibana\r\n<\/pre>\n<p>Vous devriez pouvoir lancer un navigateur et voir elasticsearch et kibana.<\/p>\n<p><b>Pour elasticsearch<\/b><\/p>\n<pre lang=\"bash\">\r\nlocalhost:9200 in web browser -- should return status code 200<\/pre>\n<p><b>Pour Kibana<\/b><\/p>\n<pre lang=\"bash\">\r\nlocalhost:5601 in web browser -- should display Kibana UI.\r\n<\/pre>\n<h4>Analyse des donn\u00e9es de contribution de la FEC (Federal Election Commission)<\/h4>\n<p>Voil\u00e0 en substance ce que le tableau de bord que vous devez obtenir.<\/p>\n<p>A l&#8217;aide de ce dashboard et des informations ainsi mis en forme, l&#8217;article concentre son analyse d&#8217;une part sur le ph\u00e9nom\u00e8ne des Super PAC, que personne ne peut d\u00e9sormais ignorer tant leur influence a \u00e9t\u00e9 grande et souterraine dans la derni\u00e8re campagne pr\u00e9sidentielle am\u00e9ricaine et d&#8217;autre part sur la possibilit\u00e9 de voir qui sont les riches donateurs dans de nombreuses villes am\u00e9ricaines. L&#8217;auteur de l&#8217;article est d&#8217;Hoboken, New Jersey, connu aussi pour \u00eatre la ville de Franck Sinatra !<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/flaven.fr\/wp-content\/uploads\/2016\/12\/elasticsearch_2_dashboard.jpg\" alt=\"ELK, elasticsearch, logstash, kibana - Exemples de data analysis \u00e0 l'aide de ELK\" width=\"500\" height=\"684\" \/><br \/>\n<i>Source : <a href=\"https:\/\/www.elastic.co\/blog\/kibana-4-and-civic-hacking-investigating-campaign-contributions\" target=\"_blank\">https:\/\/www.elastic.co\/blog\/kibana-4-and-civic-hacking-investigating-campaign-contributions<\/a><\/i><\/p>\n<p>Certes, son analyse n&#8217;est sans doute pas ultra pertinente d&#8217;un point de vue politique, pour sa d\u00e9fense l&#8217;auteur Peter Kim est un ing\u00e9nieur dont la fonction est avant tout de promouvoir l&#8217;utilisation d&#8217;elasticsearch.<br \/>\nUne le\u00e7on \u00e0 retenir n\u00e9anmoins de cette exercice, c&#8217;est l&#8217;incroyable usage qui peut-\u00eatre fait de ELK pour l&#8217;analyse d&#8217;un grand volume de donn\u00e9es que celles-ci soient structur\u00e9s ou d\u00e9structur\u00e9es ou les deux.<\/p>\n<p>En conclusion, si vous souhaitez tirer un certain nombre d&#8217;indicateurs sur un grand volume de donn\u00e9es mieux vaut s&#8217;\u00e9quiper de ELK. A ce propos, les gens de Cambridge Analytica qui ont oeuvr\u00e9s pour l&#8217;\u00e9lection de Trump l&#8217;ont d&#8217;ailleurs bien compris, rien qu&#8217;\u00e0 voir le produit &#8220;Cambridge Analytica&#8217;s Data Dashboard&#8221;, s\u00e9duisant en diable.<\/p>\n<p><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter\" src=\"https:\/\/flaven.fr\/wp-content\/uploads\/2016\/12\/elasticsearch_3_dashboard_cambridgeanalytica.jpg\" alt=\"ELK, elasticsearch, logstash, kibana - Exemples de data analysis \u00e0 l'aide de ELK\" width=\"640\" height=\"480\" \/><br \/>\n<i>Source : <a href=\"https:\/\/cambridgeanalytica.org\/datadashboard\" target=\"_blank\">https:\/\/cambridgeanalytica.org\/datadashboard<\/a><\/i><\/p>\n<h2>En savoir plus<\/h2>\n<ul>\n<li>Step-by-Step Setup of ELK for NetFlow Analytics<br \/><a href=\"http:\/\/blogs.cisco.com\/security\/step-by-step-setup-of-elk-for-netflow-analytics\" target=\"_blank\">http:\/\/blogs.cisco.com\/security\/step-by-step-setup-of-elk-for-netflow-analytics<\/a><\/li>\n<li>ELK &#8211; LOGSTASH WITH ELASTICSEARCH<br \/><a href=\"http:\/\/www.bogotobogo.com\/Hadoop\/ELK\/ELK_ElasticSearch_Logstash.php\" target=\"_blank\">http:\/\/www.bogotobogo.com\/Hadoop\/ELK\/ELK_ElasticSearch_Logstash.php<\/a><\/li>\n<li>Elasticsearch in 5 minutes<br \/><a href=\"http:\/\/www.elasticsearchtutorial.com\/elasticsearch-in-5-minutes.html\" target=\"_blank\">http:\/\/www.elasticsearchtutorial.com\/elasticsearch-in-5-minutes.html<\/a><\/li>\n<li>ElasticSearch 101 \u2013 a getting started tutorial<br \/><a href=\"http:\/\/joelabrahamsson.com\/elasticsearch-101\/\" target=\"_blank\">http:\/\/joelabrahamsson.com\/elasticsearch-101\/<\/a><\/li>\n<li>How to Install Elasticsearch on Mac OS X<br \/><a href=\"https:\/\/chartio.com\/resources\/tutorials\/how-to-install-elasticsearch-on-mac-os-x\/#modify-your-bashprofile-variables\" target=\"_blank\">https:\/\/chartio.com\/resources\/tutorials\/how-to-install-elasticsearch-on-mac-os-x\/#modify-your-bashprofile-variables<\/a><\/li>\n<li>How to Setup Logstash on Linux with ElasticSearch, Redis, Nginx<br \/><a href=\"http:\/\/www.thegeekstuff.com\/2014\/12\/logstash-setup\/\" target=\"_blank\">http:\/\/www.thegeekstuff.com\/2014\/12\/logstash-setup\/<\/a><\/li>\n<li>Kibana 4 Tutorial \u2013 Part 1: Introduction<br \/><a href=\"https:\/\/www.timroes.de\/2015\/02\/07\/kibana-4-tutorial-part-1-introduction\/\" target=\"_blank\">https:\/\/www.timroes.de\/2015\/02\/07\/kibana-4-tutorial-part-1-introduction\/<\/a><\/li>\n<li>Kibana 4 &#038; Civic Hacking: Investigating Campaign Contributions<br \/><a href=\"https:\/\/www.elastic.co\/blog\/kibana-4-and-civic-hacking-investigating-campaign-contributions\" target=\"_blank\">https:\/\/www.elastic.co\/blog\/kibana-4-and-civic-hacking-investigating-campaign-contributions<\/a><\/li>\n<li>Elasticsearch\u200bClient<br \/><a href=\"https:\/\/packagecontrol.io\/packages\/ElasticsearchClient\" target=\"_blank\">https:\/\/packagecontrol.io\/packages\/ElasticsearchClient<\/a><\/li>\n<li>Installing Kibana for Elasticsearch on OS X<br \/>\n<br \/><a href=\"https:\/\/www.youtube.com\/watch?v=psNH33pcGBo\" target=\"_blank\">https:\/\/www.youtube.com\/watch?v=psNH33pcGBo<\/a><\/li>\n<li>Visualizing Logs Using ElasticSearch, Logstash and Kibana<br \/><a href=\"https:\/\/www.youtube.com\/watch?v=Kqs7UcCJquM\" target=\"_blank\">https:\/\/www.youtube.com\/watch?v=Kqs7UcCJquM<\/a><\/li>\n<li>ELK Stack on Mac OS X Yosemite (excellent)<br \/><a href=\"http:\/\/www.websightdesigns.com\/wiki\/ELK_Stack_on_Mac_OS_X_Yosemite\" target=\"_blank\">http:\/\/www.websightdesigns.com\/wiki\/ELK_Stack_on_Mac_OS_X_Yosemite<\/a><\/li>\n<li>Installing Kibana for Elasticsearch on OS X<br \/><a href=\"http:\/\/codingexplained.com\/operating-systems\/mac\/installing-kibana-for-elasticsearch-on-os-x\" target=\"_blank\">http:\/\/codingexplained.com\/operating-systems\/mac\/installing-kibana-for-elasticsearch-on-os-x<\/a><\/li>\n<li>Elasticsearch: Getting Started<br \/><a href=\"https:\/\/www.elastic.co\/webinars\/getting-started-elasticsearch\" target=\"_blank\">https:\/\/www.elastic.co\/webinars\/getting-started-elasticsearch<\/a><\/li>\n<li>Creating a Kibana dashboard of Twitter data pushed to Elasticsearch with NiFi<br \/><a href=\"https:\/\/community.hortonworks.com\/articles\/56648\/creating-a-kibana-dashboard-of-twitter-data-pushed.html\" target=\"_blank\">https:\/\/community.hortonworks.com\/articles\/56648\/creating-a-kibana-dashboard-of-twitter-data-pushed.html<\/a><\/li>\n<li>Le site officiel de logstash<br \/><a href=\"http:\/\/logstash.net\/\" target=\"_blank\">http:\/\/logstash.net\/<\/a><\/li>\n<li>Le site officiel d&#8217;elastic search<br \/><a href=\"http:\/\/www.elasticsearch.org\/\" target=\"_blank\">http:\/\/www.elasticsearch.org\/<\/a><\/li>\n<li>Le site officiel de kibana<br \/><a href=\"http:\/\/www.elasticsearch.org\/overview\/kibana\/\n\" target=\"_blank\">http:\/\/www.elasticsearch.org\/overview\/kibana\/<br \/>\n<\/a><\/li>\n<li>ElasticSearch: can&#8217;t connect via curl, weird inconsistent behavior<br \/><a href=\"http:\/\/stackoverflow.com\/questions\/27953770\/elasticsearch-cant-connect-via-curl-weird-inconsistent-behavior\" target=\"_blank\">http:\/\/stackoverflow.com\/questions\/27953770\/elasticsearch-cant-connect-via-curl-weird-inconsistent-behavior<\/a><\/li>\n<li>Cambridge Analytica, les machiavels de la data<br \/><a href=\"https:\/\/cambridgeanalytica.org\/\" target=\"_blank\">https:\/\/cambridgeanalytica.org\/<\/a><\/li>\n<li>Le blog de Peter Kim<br \/><a href=\"http:\/\/peter.mistermoo.com\/\" target=\"_blank\">http:\/\/peter.mistermoo.com\/<\/a><\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>Il existe un trio remarquable pour la lecture des fichiers de logs. Triptyque qui rend intuitif et parlant un certain nombre d&#8217;informations-cl\u00e9s notamment si vous&hellip; <\/p>\n<p class=\"text-center\"><a href=\"https:\/\/flaven.fr\/2016\/12\/elk-elasticsearch-logstash-kibana-exemples-de-data-analysis-a-laide-de-elk\/\" class=\"more-link\">Continue reading &rarr; <span class=\"screen-reader-text\">ELK, elasticsearch, logstash, kibana &#8211; Exemples de data analysis \u00e0 l&#8217;aide de ELK<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":10798,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"bf_ai_meta_description":"ELK stack tools for data analysis. See examples of elasticsearch, logstash, kibana in action. Achieve clear insights from log files.","bf_ai_og_title":"ELK Stack Data Analysis","footnotes":"","jetpack_publicize_message":"","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","default_image_id":0,"font":"","enabled":false},"version":2},"jetpack_post_was_ever_published":false},"categories":[3437,3454,3442,3444,3447,3448,3449,3450,3435],"tags":[2363,2362,2365,2364,606],"class_list":["post-10794","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-business-case-studies","category-journalism-writing","category-mobile-devices","category-programming-databases","category-technology-trends","category-tools-productivity","category-tutorials-how-to","category-ux-product-design","category-web-development","tag-elasticsearch","tag-elk","tag-kibana","tag-logstash","tag-storytelling"],"jetpack_publicize_connections":[],"jetpack_sharing_enabled":true,"jetpack_shortlink":"https:\/\/wp.me\/p3Vuhl-2O6","jetpack_featured_media_url":"https:\/\/flaven.fr\/wp-content\/uploads\/2016\/12\/elasticsearch_bf.jpg","_links":{"self":[{"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/posts\/10794","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/comments?post=10794"}],"version-history":[{"count":6,"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/posts\/10794\/revisions"}],"predecessor-version":[{"id":10931,"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/posts\/10794\/revisions\/10931"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/media\/10798"}],"wp:attachment":[{"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/media?parent=10794"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/categories?post=10794"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/flaven.fr\/happy-api\/wp\/v2\/tags?post=10794"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}