{"id":26661,"date":"2020-06-03T00:00:00","date_gmt":"2020-06-02T22:00:00","guid":{"rendered":"https:\/\/blexin.com\/creare-dashboard-con-elasticsearch-e-kibana\/"},"modified":"2021-06-16T18:08:21","modified_gmt":"2021-06-16T16:08:21","slug":"lets-see-how-creating-dashboards-starting-from-data-in-elasticsearch-using-kibana","status":"publish","type":"post","link":"https:\/\/blexin.com\/en\/blog-en\/lets-see-how-creating-dashboards-starting-from-data-in-elasticsearch-using-kibana\/","title":{"rendered":"How to create dashboards with ElasticSearch and Kibana"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n

In my previous articles (here the first<\/a> and the second<\/a> one), I have shown the use of ElasticSearch as a full-text search engine in e-commerce, the set and use of some advanced configurations and the creation of a products <\/code>index which contains all saved products.<\/p>\n\n\n\n

For demonstration purposes, we used the Bogus library for the dynamic generation of products and the NEST library for the CRUD on the ElasticSearch index.<\/p>\n\n\n\n

Our model Product class is defined as:<\/p>\n\n\n

\npublic class Product\n{\n\u00a0\u00a0\u00a0\u00a0public int Id { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Ean { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Name { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Description { get; set; }\n\u00a0\u00a0\u00a0\u00a0public Brand Brand { get; set; }\n\u00a0\u00a0\u00a0\u00a0public Category Category { get; set; }\n\u00a0\u00a0\u00a0\u00a0public Store Store { get; set; }\n\u00a0\u00a0\u00a0\u00a0public decimal Price { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Currency { get; set; }\n\u00a0\u00a0\u00a0\u00a0public int Quantity { get; set; }\n\u00a0\u00a0\u00a0\u00a0public float Rating { get; set; }\n\u00a0\u00a0\u00a0\u00a0public DateTime ReleaseDate { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Image { get; set; }\n\u00a0\u00a0\u00a0\u00a0public List<review> Reviews { get; set; }\n}\n<\/pre><\/div>\n\n\n
where Brand, Category, Store, Review, and User classes are respectively:<\/p>\n\n\n
\npublic class Brand\n{\n\u00a0\u00a0\u00a0\u00a0public int Id { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Name { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Description { get; set; }\n}\n\u00a0\npublic class Category\n{\n\u00a0\u00a0\u00a0\u00a0public int Id { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Name { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Description { get; set; }\n}\n\u00a0\npublic class Store\n{\n\u00a0\u00a0\u00a0\u00a0public int Id { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Name { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Description { get; set; }\n}\n\u00a0\npublic class Review\n{\n\u00a0\u00a0\u00a0\u00a0public int Id { get; set; }\n\u00a0\u00a0\u00a0\u00a0public short Rating { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string Description { get; set; }\n\u00a0\u00a0\u00a0\u00a0public User User { get; set; }\n}\n\u00a0\npublic class User\n{\n\u00a0\u00a0\u00a0\u00a0public int Id { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string FirstName { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string LastName { get; set; }\n\u00a0\u00a0\u00a0\u00a0public string IPAddress { get; set; }\n\u00a0\u00a0\u00a0\u00a0public GeoIp GeoIp { get; set; }\n}\n<\/pre><\/div>\n\n\n
GeoIP is a class from the NEST library used for geographic data.<\/p>\n\n\n\n
The next step is to create a dashboard that can display the products, and the performed research carried out on them to elaborate more or less advanced statistics.<\/p>\n\n\n\n
Rather than developing a custom solution with certain time-consuming efforts, we decided to use Kibana<\/a>. It deals with a front end application that is part of the ElasticSearch stack that allows us to view data and searches for all indexed data and track the load of queries.<\/p>\n\n\n\n
Kibana can also be used for monitoring, managing, and securing the same stack.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
To install and configure it just go om the page https:\/\/www.elastic.co\/downloads\/kibana<\/a>. Here we can find the installers for all platforms, and download the one suitable for our needs.<\/p>\n\n\n\n
Once downloaded and unzipped in a given folder, in our case C:\\ElasticSearch\\Kibana, we open the config\/kibana.yml file and set the elasticsearch.hosts parameter to point to our instance of ElasticSearch (for local versions http:\/\/localhost:9200<\/a>):<\/p>\n\n\n\n
elasticsearch.hosts: [\"http:\/\/localhost:9200\"]<\/code><\/p>\n\n\n\n
Let\u2019s run bin\/kibana.bat and open the http:\/\/localhost:5601<\/a> from a browser page. The page we get is the following:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Kibana interface is divided into several sections.<\/p>\n\n\n\n
In the open source version there are: Discover (data interactive exploration), Visualize (data analysis in graphs, tables, tags), Dashboards (complex data views), Canvas (documents creation), Maps (georeferenced data analysis), Dev Tools (tools to process and analyze queries) and Management (index and cluster management).<\/p>\n\n\n\n
We can also install the X-Pack plugin to use the Graph and Monitoring sections.<\/p>\n\n\n\n
Let’s go to the Management -> Elasticsearch -> Index Management section to verify that the Elasticsearch indexes have been correctly detected:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
We can find our products<\/code> index and verify its mapping and matching with our data model:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
To create a Kibana index, just go to the Management section -> Kibana -> Index Patterns and type a text that allows you to link the new index to one or more ElasticSearch indexes.<\/p>\n\n\n\n
In our case, we type products<\/code>, so as to create our Kibana index.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Once created the index, in the Discover section, it is possible to filter the data by date or by one or more fields:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Using the search bar, we can query between products using the KQL language (Kibana Query language), which allows you to easily query using the autocomplete. For example, we could type:<\/p>\n\n\n\n
category.name : Games AND rating > 0.5<\/code><\/p>\n\n\n\n
to know all the products in the Games category with a rating higher than 0.5.<\/p>\n\n\n\n
You can select some field and add them to Selected fields, in order to have a custom result view.<\/p>\n\n\n\n
Once added the index and verified the correctness of the queries, we can create new data views.<\/p>\n\n\n\n
The visualizations consist of various types of graphs (bar, cakes), tables, indicators, metrics, and tag clouds. Of course, they support data aggregations.<\/p>\n\n\n\n
In the Visualize section, we can create a new data visualization by using graphs.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
For statistical purposes, we create some graphs of products grouped by category, brand, using simple vertical bar graphs. We get results similar to:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
You can also add filters to this view. For example, in our case, we want to see only products available in the store, so we type quantity > 0<\/code> in the filter bar. We click then on the Save button to save our view.<\/p>\n\n\n\n
Another useful visualization is that of products by price range. In this case, we define buckets in the price field and use them for a pie chart. Let\u2019s divide the products by the following price ranges:<\/p>\n\n\n\n
0     50
50    100
100   200
200   400
400   800
800<\/code><\/p>\n\n\n\n
We can get a result similar to the following:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
We can also add some sub-buckets to have aggregated data and a nested visualization.<\/p>\n\n\n\n
In the Maps section, we can use Elastic Maps (multilayer maps) to show georeferenced data.<\/p>\n\n\n\n
We can click on the Add layer to add data to our index by selecting which field contains georeferences info (in our case geoIp.location<\/code>).<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Once we created all the views needed, we move on to our first Dashboard. This last is a set of views, searches, and maps, usually updated in real-time, which provides high-level information on the index data.<\/p>\n\n\n\n
In the Dashboard section, let\u2019s click on Create a new dashboard, then to Add and select the created views:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Let\u2019s add them all and arrange them on the dashboard layout. We can obtain such a result:<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Dashboards can be filtered through KQL queries, and the view is always dynamic. We can also share and integrate them within our Web applications through iframes. If we click on Share -> Copy Iframe Code we will get such a link:<\/p>\n\n\n\n
<iframe src=\"http:\/\/localhost:5601\/app\/kibana#\/dashboard?embed=true\" height=\"600\" width=\"800\"><\/iframe><\/code><\/p>\n\n\n\n
Another interesting feature of Kibana is Canvas. It deals with a tool for viewing and presenting data that show real-time data and combines them with colors, images, and text to create dynamic views.<\/p>\n\n\n\n
In the Canvas section, let\u2019s click on Create workpad and start adding the metrics. In our case, we set parameters for the products and available items, brands, and categories, a pie chart for the brand-category pair and an average of the prices of the items by brand.<\/p>\n\n\n\n
\"\"<\/figure>\n\n\n\n
Once created, the workpad can be shared as a JSON file or downloaded as a PDF report.<\/p>\n\n\n\n
Other interesting features are:<\/p>\n\n\n\n