July 08, 2009
Our recent update to Scout featured a revised UI, more functionality, and a new Scout Agent. While it’s easy to see the changes in the UI, a lot of the work conducted by the agent happens beneath the surface.
The Scout Agent, which is installed on a server you wish to monitor, was kind enough to sit down and walk me through its DNA (note that the ability to answer human questions is currently not available in the most recent release).
I’m just a plain-old Ruby gem that you can install on any Linux-based server (
sudo gem install scout_agent).
Yes, I’m a daemon. And yes, Ruby, like many programming languages, can leak memory when run for a long period of time.
My strategy for preventing memory leaks is simple: I do real work, like running plugins, in a separate short-lived process. I
fork(), do whatever, and
exit() so the OS can clean up any mess.
My work is divided into 2 main processes and several short-lived processes:
The reason for this division of labor? The real work is executed by the mission processes, which are short-lived. By offloading the work to such processes, the potential for degrading performance and a plugin’s execution raising an exception and killing me off is greatly reduced.
It’s easier to write 200 lines of bug-free code than 3000. The 200 LOC (my lifeline) keeps the rest alive.
I typically use between 15-20 MB of memory.
Every 3 minutes, I wake up and contact the Scout server and grab the plugins I’m supposed to run. I then run each plugin and report the data back to the server.
I validate the SSL certificate of the Scout server. I won’t send any data to or take any data from a source that fails this validation. This prevents a man-in-the-middle attack.
The server caches the plugin code when you install the plugin. The code isn’t updated unless you choose to do so. Most plugins have just a few lines of code – usually less than 50 – so it’s easy to do a review. All of the plugins listed in Scout’s plugin directory are also maintained exclusively by the Scout team.
The agent doesn’t run plugins as the root user – by default, it uses the daemon user. You can change the user in the /etc/scout_agent.rb file.
Yes – I record every plugin run in my log file, which is typically located in