Graphite is a very popular enterprise monitoring tool. This article will address the common issues that occur while setting up a Graphite instance, and how they can be avoided. We will assume readers have already become acquainted with Graphite, but if you’re interested to learn about the basics of Graphite, check out our articles on the Architecture and Concepts of Graphite and the Installation and Setup before reading this article. For this article, the key terms you need to know are:
We will also evaluate the benefits and costs of setting up your own instance of Graphite vs using a Hosted Graphite service such as MetricFire. As operations scale, it can become too time-consuming to run your own Graphite instance, making a hosted version much more feasible.
As you’re going through this article, try out your own Hosted Graphite with our free trial account. You will be able to send metrics directly and test to see how it works.
In our installation tutorial, we used docker images of Graphite to run and demo the Graphite application. However, to achieve the best performance, Graphite needs to be installed on a production machine running either CentOS / Debian. This requires following a lot of manual steps to get the Graphite up and running.
So, let’s get started. In this example, I will be using Ubuntu 18.04 to get the Graphite up and running.
sudo apt-get update
sudo apt-get install graphite-web graphite-carbon
During the installation process, the prompt will appear asking if you want to keep the Carbon database if you ever decide to uninstall and re-install. You can choose no for this prompt.
Although technically now you have installed Graphite and Carbon, there are various caveats which we are going to address now one by one.
The Graphite web application uses SQLite3 database files by default to store its data. This is not at all recommended for production systems since SQLite3 is not a full fledged database system. Hence, we will install and configure PostGreSQL to let Graphite store the data there.
sudo apt-get install postgresql libpq-dev python-psycopg2
This will install PostGreSQL to which Graphite will connect to and store the data.
Now, we will create the user Graphite and database Graphite in the database.
Now, we need to tell Graphite to use our new database server settings instead of the default database which is SQLite. Change the values in /etc/graphite/local_settings.py from the following default values to the new database settings we just created.
Run the command “sudo graphite-manage migrate” to let Django create tables in the new database. You should see the following output:
If your production server gets restarted for some reason, the Graphite and Carbon won’t start automatically. For it to start automatically, we need to make some changes in /etc/default/graphite-carbon: Change the value of CARBON_CACHE_ENABLED to true.
This will ensure that Graphite and Carbon are started automatically on boot.
By default, carbon doesn’t do any log rotation. That means, the hard disk is at risk of filling up unless you clean your log files manually or enable log rotation.
1. Edit the file /etc/carbon/carbon.conf
2. Change the value of ENABLE_LOGROTATION to True
To get the graphite web interface up and running, we need to install apache
sudo apt-get install apache2 libapache2-mod-wsgi
To enable the graphite on Apache web server, run the following commands
Once this is done, if you go to http://localhost, you should finally see Graphite Web application:
If you face any issue while setting up Graphite and Carbon for production:
Most of the time, the problem you are facing has been faced by someone else previously as well.
If you have made it to the end of article and are able to successfully run Graphite, congratulations. If you face any issue, there are numerous troubleshooting articles on the Internet to get it up and running, which we will not replicate here. You can also reach out to MetricFire and see if a hosted version of Graphite is better suited
In the rest of the article, we will analyze the benefits of using Hosted Graphite such as MetricFire vs using your own Graphite. MetricFire helps in the following ways:
In the article above, we haven't really touched scaling Graphite, which will be required at any production grade deployment. It requires running multiple instances of Carbon in the same host and also across multiple hosts in a cluster. Setting that up isn’t trivial and requires a lot of administrative manpower and monitoring.
Hosted Graphite offers automatic backup of data to make sure if something goes wrong, data can be recovered quickly and business isn’t affected. Deploying a custom backup solution in self-hosted Graphite is an expensive exercise and requires deploying additional manpower and storage to take care of backups.
Hosted Graphite provides year round support and automated monitoring anywhere in the world.
Hosted graphite provides high security using protocols such as HTTPS and TLS to transmit data packets. Configuring self-hosted Graphite to provide such security requires a lot of custom configuration to get it up and running.
Graphite consists of a lot of moving parts such as Python, Django, Apache Web Server and database engine. In order to run it at the optimum levels, all these components need to be updated at regular intervals. Hosted Graphite solves these problems for you by applying the latest updates for these components in the background.
As well the actual technology of Hosted Graphite is slightly different than regular Graphite. The developers have added to the Graphite project to make Hosted Graphite even easier for the users. We outline this topic in the article Hosted Graphite isn't Graphite.
We have seen the process of setting up self hosted Graphite. It doesn’t cover advanced use cases like scaling, security and backups. Such features are only provided by hosted Graphite. Self hosted Graphite requires administration and specialized engineers who are experts in Linux Administration, Graphite, python, Django, Apache Web Server and PostGreSQL. For most setups, the cost of running self hosted Graphite exceeds the cost of using hosted Graphite.
To try Hosted Graphite, log in to MetricFire's Hosted Graphite free trial, and start sending metrics today. You can also reach out to us by booking a demo, we're always happy to jump on a call and talk about your company's monitoring needs.