How Scout Works
- How do I use the Scout Agent?
- What are the Scout Agent system requirements?
- What impact does the Scout Agent have on performance?
- How does the Scout Agent execute plugins?
- What happens if a new version of a plugin is released?
- How does Scout approach security?
- Is Scout open source?
- How long does it take for data to appear on my Scout account?
- How do I run the Scout Agent behind a firewall?
- How do webhook calls from Scout work?
- How do I setup Scout to run using cron?
- How do I setup Scout to run using the system-wide crontab file?
- How do I setup Scout to run using my current user's crontab file?
- Scout doesn't run in cron. I'm seeing a "Permission denied" error.
- Scout doesn't run in cron. I'm seeing a "ruby: command not found" error.
- I'm still having problems running Scout through cron.
- What is Scout's approach to security?
- How does Scout's architecture make it secure?
- Can I place sensitive plugin settings (like passwords) locally on the server?
- Can plugin code be updated on my server without my knowledge?
- Are there any additional security measures I need to take with Scout?
- How do I Solve 'scout: command not found'?
- Scout Prints the Error: 'no such file to load -- net/https'
- Scout Prints the Error: 'no such file to load -- openssl'
- Updating Rubygems
- Common Ruby Ubuntu problems
- Scout Prints the Error: 'zlib(finalizer): Zlib::GzipWriter object must be closed explicitly'
- Does the Scout Agent generate a log file?
- I'm stuck. How do I get help?
Inserting Report Data Into Scout
- Can I associate report data with an arbitrary time?
- Scout isn't graphing fields it should be (or vice versa). Does report data need to fit a certain format?
- Is there a maximum size for a report field name?
- How can I automatically remove instances from Scout when they are shutdown?
- Can I reduce the no data alerting threshold to less than 30 minutes?
- I'm having problems authenticating to retrieve RSS Feeds.
- Will Scout start automatically when my server is rebooted?
- How do I uninstall Scout?
- What happened to cloud keys?
- What happened to plugin copy/paste
How Scout Works
How do I use the Scout Agent?
Installing the agent is just a simple Ruby gem install:
At this point, you should be all set to run the agent. Just add Scout to your crontab file so it runs every minute (see our section on Cron):
With the agent running, you should be able to log into your account on scoutapp.com to setup your list of plugins and see the agent delivering data.
What are the Scout Agent system requirements?
The Scout Agent has been tested on the following operating systems: Linux, Apple OSX, and Solaris. Ruby 1.8.6 or higher and Rubygems 1.3.1 or higher must be installed.
What impact does the Scout Agent have on performance?
The Scout Agent is very lightweight. It only uses resources when it runs through Cron. It isn't a long-running process so it won't leak memory. Most plugins scrape basic Linux commands, so the plugins themselves usually don't load heavy libraries. On a typical system, the agent usually consumes less than 15 MB of memory when running.
How does the Scout Agent execute plugins?
The Scout Agent retrieves the plugin plan from the scoutapp.com server when it runs. The plugin plan includes the plugin code plus any plugin-specific options you've set. The agent then executes the code, with the provided options, and reports back the results to Scout.
Plugins are not manually installed on the agent. You configure all plugins on scoutapp.com. Once the Scout Agent is installed, you're done!
What happens if a new version of a plugin is released?
Scout will never update the code running on your server without your explicit instructions to do so. If a new version of a plugin is released, the plugin won't be updated with the new code. If you wish to update a plugin to a new version, just click the "Update Code" button on the Plugin Settings page.
Is Scout open source?
The Scout agent is completely open source. The gem is a normal Ruby gem, open for development, available on GitHub and distributed under the MIT and/or Ruby License (whichever you prefer). We share a collection of open-source Scout Plugins surrounded and fostered by a community that encourages branching, fixes, and general open-ness. Scout plugins can be accessed via GitHub.
The Scout Server -- which handles the data collection, analysis, trending, and notifications -- is not open-source. We maintain the server, and keep all your data safe and sound.
How long does it take for data to appear on my Scout account?
Typically it takes less than 5 minutes for data to appear on your account.
How do I run the Scout Agent behind a firewall?
Scout communicates via HTTP and HTTPS. All communication is outgoing, so no incoming ports need to be open. Ensure ports 80 and 443 are open for outgoing communication.
How do webhook calls from Scout work?
Scout can perform an HTTP POST (a webhook call) when a trigger generates an alert. See the Developer Resources page on webhooks for details.
How do I setup Scout to run using cron?
You need to configure Scout to run automatically after it is installed. Cron is the easiest way to have Scout run at a scheduled interval on a Linux, Solaris or BSD system.
Cron jobs are configured using the crontab file (also called the cron table). Each user can have their own crontab file, and there is also a system-wide crontab file located here:
If you have root access, find out: "How do I setup Scout to run using the system-wide crontab file?"
If you do not have root access, find out "How do I setup Scout to run using my current user's crontab file?"
How do I setup Scout to run using the system-wide crontab file?
As the root user (or using sudo), open the crontab file for editing:
We will need to add a line to the bottom of the file with our cron job.
To run the Scout agent every minute, we'll use the following format:
The above cron job runs every minute as the deploy user.
Save the file, and it will automatically run every minute.
How do I setup Scout to run using my current user's crontab file?
We will need to use the crontab command to edit our current user's crontab.
Run the crontab edit command:
If you do not have any jobs already setup, you should see a blank file. If you get an error, see this Unix Crontab Reference for more help.
To run the Scout agent every minute, we'll use the following format:
The above cron job runs every minute as the current user.
Save the file, and it will automatically run every minute.
Scout doesn't run in cron. I'm seeing a "Permission denied" error.
Scout probably doesn't have access to the data file it needs to store information locally. You may be scheduling the job to run as a different user than the one you installed Scout with, and that user may not be able to write to the data file.
You can get around this by specifying a path to a data file (doesn't matter if it exists yet) that the user in the cron job has access to using the "-d option":
The above job stores data to the file located at
Scout doesn't run in cron. I'm seeing a "ruby: command not found" error.
Scout can't find the Ruby executer. Your cron path may not contain the full path to Ruby.
Be sure to update your
/etc/crontab if you have installed Ruby in an alternate location.
For example, if Ruby is installed in "
PATH line my look like:
If you're using RVM, read about using RVM and cron in production.
I'm still having problems running Scout through cron.
Here are a few tips for debugging Scout in the crontab file. We can tell Scout to log to a file:
Using the -v option tells it to be verbose. Using the -l debug option tells it to use the debugging log level.
For more help on cron and the crontab file, see this Unix Crontab Reference.
How do I setup Scout to run using launchd?
Scout can be configured to run at a scheduled interval using launchd, or as it sometimes called, launchctl or LaunchDaemon. Launchd can be used by Mac OS X systems to run Scout regularly.
To create a LaunchDaemon, simply create a file, named com.scoutapp.ScoutAgent.plist with the following properties:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>com.scoutapp.ScoutAgent</string> <key>ProgramArguments</key> <array> <string>[PATH TO RUBY]</string> <string>[PATH TO SCOUT]</string> <string>[YOUR KEY]</string> <string>-d [DATA FILE]</string> </array> <key>StartInterval</key> <integer>[INTERVAL IN SECONDS]</integer> </dict> </plist>
Here's an example file with all the details filled in:
<?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>com.scoutapp.ScoutAgent</string> <key>ProgramArguments</key> <array> <string>/usr/local/bin/ruby</string> <string>/usr/local/bin/scout</string> <string>8c6dc00d-af3a-4606-a384-85f276ecbbb5</string> <string>-d /tmp/scout_data.yml</string> </array> <key>StartInterval</key> <integer>60</integer> </dict> </plist>
Next, copy this file to /Library/LaunchDaemons :
To install it, you can simply restart the system, or load it instantly:
The daemon will run every minute (60 seconds) using the /tmp/scout_data.yml file for storage.
Scout doesn't run in launchd. I'm not seeing any errors.
In order to view errors produced by launchd, you'll need to visit the Console, or tail the
Launchd logs messages and problems running the Scout Client there.
What is Scout's approach to security?
Security is important to us, and we are committed to ensuring that Scout meets the same high standards you maintain for your own servers.
First, we ensure that only our server is sending you plugins. The agent validates the SSL certificate of the Scout server and will only send and receive data if the certificate is validated. This makes a man-in-the-middle-attack practically impossible.
After that, we ensure our server doesn't send you bad plugins. First, we code-review each plugin that is accepted into our directory. We also don't think you should just take our word for it, which is why we show you the code when you add a plugin. This isn't nearly as it involved as it sounds since the majority of plugins are fairly short and straight forward. Second, plugin code is signed via a private key. The agent only executes a plugin if the plugin signature is validated with the associated public key.
In addition to the plugins being available to you, we also keep the agent and plugins open source. One of our major reasons for doing that is to ensure that your security team has full access to the code we are asking you to use on your server.
How does Scout's architecture make it secure?
- You install the Scout Agent (which is a Ruby gem) on your server.
- The agent connects over https (always) through a 256-bit secure, encrypted connection (the same encryption your bank uses).
- Scout never logs in to any of your servers.
- All communication is initiated by the agent — it is impossible for a 3rd party to initiate any operations by contacting the Scout agent.
- The agent downloads a pre-loaded plugin plan, consisting only of plugins of your choosing, so it cannot run plugins you didn’t explicitly authorize.
- The server also uses that same secure encryption for all communication. Individual accounts are protected.
- Agent keys (uniquely generated) can be revoked at any time, disabling the agent.
Can I place sensitive plugin settings (like passwords) locally on the server?
Yes. You can tell Scout to store the settings locally in a
plugins.properties file. There are two steps:
plugins.propertiesfile in the .
scoutdirectory on your server (by default,
/home/USER_RUNNING_SCOUT/.scout/plugins.properties). Example contents:# this is a comment line
- In the web interface, tell Scout look up the configuration:
- The lookup keys are completely up to you - the only requirement is
that what you specify in the web interface must match the keys you supply
- Multiple plugins can use the same lookup key, so, for example, if
you have two plugins using a MySQL password, they can both reference
the same key in
- If a plugin in a role uses lookup keys, you'll have to copy
plugins.propertiesto each server in the role.
- When setting up
plugins.properties, we recommend running Scout manually and watching the debug output. It will confirm that it's using
plugin.propertiesas you expect. To run manually with debug output:scout YOUR_KEY -v -ldebug --f
- When attepting to install a gem: ERROR: While executing gem ... (Gem::GemNotFoundException)
- ... or Could not find elif (> 0) in any repository
- When running the Rails Analyzer plugin: Could not find RubyGem rspec (>= 1.2.4) (Gem::LoadError)
- Bulk operations: instead of adding plugins and triggers directly to servers, configure roles and add servers to roles.
- Roles are "active": if you update a role (say by adding a plugin or a trigger), all the servers in that role are automatically updated to reflect the changes.
- A server can belong to multiple roles. It will have the superset of all plugins in all its roles.
- Account-wide keys: no need to provision keys for new servers. Reuse the same 40-character account key in the crontab across all your servers. (note - your existing keys will still work!)
- Specify roles via the crontab: optionally, you can pass a command-line argument to the scout agent to specify the roles it should belong to.
- Friendly with Chef: we provide an official chef recipe for roles-enabled server configuration.
Can plugin code be updated on my server without my knowledge?
No. Even if new code is checked into the plugin repository for a plugin you have installed, your plugin will not be updated unless you explicitly upgrade it.
Are there any additional security measures I need to take with Scout?
As long as you are running approved plugins from the Scout plugin repository, there is nothing more you need to do. If you are running plugins from outside our repository, treat them like any other code you introduce into your production environment: make sure it comes from a credible source, and review the source code if you have an doubts. Most plugins are relatively few lines of code and easy to review.
How do I Solve 'scout: command not found'?
Assuming you've successfully installed the Scout gem, this indicates a problem with your path. Specifically, your path doesn't include executables installed via RubyGems. To locate the Scout executable, run:
The EXECUTABLE DIRECTORY contains the Scout executable. So in this case, you could run scout with:
For a longer-term solution, add the value of EXECUTABLE_DIRECTORY to your PATH. Depending on your system configurations, you may need to set PATH separately for crontab. Or, just use the full path to the scout executable in your crontab.
Scout Prints the Error: 'no such file to load -- net/https'
This happens most often on Debian installs, because Debian does not install LibOpenSSL by default with Ruby. To fix:
Advanced: if you are compiling Ruby from scratch, make sure you install (through aptitude) openssl libssl libssl-dev before compiling Ruby.
Scout Prints the Error: 'no such file to load -- openssl'
We've seen this issue on Centos installs. To fix:
There are variety of issues that are resolved by upgrading the latest version of Rubygems. These issues can arise as:
To fix, first try:
The gem --version command should return at least 1.3.4. If this doesn't succeed, try the following:
sudo gem install -v 1.3.4 rubygems-update
sudo gem update --system
if your current version of Rubygems is very old, this approach may not work either. Ubuntu 8 (Hardy) in particular gives you an ancient version of rubygems through apt-get. So, first, uninstall any existing rubygems installation:
sudo apt-get purge libgems-ruby1.8
mkdir src; cd src
tar xzf rubygems-1.3.5.tgz
sudo ruby setup.rb
sudo ln -s /usr/bin/gem1.8 /usr/bin/gem
sudo gem update --system
Common Ruby Ubuntu problems
Ruby on Ubuntu can present missing library errors. The blog post Troubleshooting common Ruby Ubuntu problems solves a number of these problems.
zlib(finalizer): Zlib::GzipWriter object must be closed explicitly
If you see the following error when running the Scout agent:
zlib(finalizer): the stream was freed prematurely.
Ensure you are using version 5.1.3 of the Scout Agent or higher. To update:
If you still receive the error, run the agent in debug mode:
If you see the following error message:
There is likely a conflict between the
Install json_pure and json so that both versions are in sync:
gem install json_pure
Does the Scout Agent generate a log file?
The agent stores the log output from the most recent run in /home/[USER]/latest_run.log. If you wish to log all runs, enable verbose logging when running the agent and store the output in a file:
The above line saves the output of the
scout command to the
I'm stuck. How do I get help?
Contact us at firstname.lastname@example.org. We usually respond in a couple of hours during the business day. Please provide your operating system, account name and any error messages you received.
The basics of server roles
Roles let you install and manage a set of plugins across multiple servers:
Can I choose which roles show on my homepage?
Yes. To hide a role that's currently shown, hover over the role name, click the wrench, and choose hide. To show a role that's currently hidden, scroll to the bottom of your homepage, click "show hidden roles," then hover over the name, click the wrench, and choose show.
Is it ok to create a role with no plugins in it?
Absolutely. Sometimes you want to create a visual grouping for your homepage without assigning any plugins. You can create a role for this and assign servers without installing any plugins.
Can I assign roles to a server from the command line?
Yes, you can specify roles via the Scout agent with the `-r` flag. If you specify roles via the agent, it will override any roles selections you've made through the UI. Be careful not to inadvertently remove roles (and therefore delete plugins and data) when you start specifying roles via the agent.
When specifying a role through the command line, use the role's "agent name" - a downcased version with special characters removed. The agent name is displayed on the role's sidebar.
Do all plugins have to be in roles?
Plugins don't have to be in roles -- you can install a plugin directly on a server without installing it through a role. This is useful for trying out a plugin before committing it to a broader group of servers.
Can I add or modify a trigger on just one instance of a plugin that's in a role?
No. If you need an additional trigger beyond what's specified in the role, you can install a second instance of the plugin on just that server, and add the trigger to that. If you need different triggers, you can split the plugin into multiple roles and manage them separately -- see the Advanced section below.
Move plugin template to a different role
From a plugin template page, click "Move plugin template" on the right-hand side. When you move a plugin template from Role A to Role B, two things happen:
- the plugin is installed on any servers in Role B that aren't also in Role A.
- the plugin is deleted from any servers in Role A that aren't also in Role B. Be careful when moving plugin templates -- you can lose data.
An example of when to use "move plugin template": you have 10 app servers, and you've created a temporary role for two of them with a plugin you are testing. Now, you want to install the plugin on the remaining 8 servers, without losing the data you've already gathered. "Move plugin template" will accomplish this: the data on the two servers will be preserved, and new plugin instances will be installed on the remaining 8 app servers as part of the plugin move.
Split plugin template into multiple roles
From a plugin template page, click "split plugin template" on the right-hand side. You'll be given the opportunity to select the roles to move the plugin template into. The "move-to" roles must contain a subset of the servers in the move-from role.
An example of when to use this: you have a plugin in "All Servers," that needs different configurations on your app role vs your db role. For simplicity, assume you have five servers total, three of which are app servers and two of which are db servers. When you split the plugin from "All servers" to app and db, you will have two separate plugin templates you can manage (one in app, and one in db), with different triggers and different settings.
Splitting a plugin gives you more granular control and the expense of centralized configuration. After splitting a plugin into two different roles, if you need to make the same change to all instances of the plugin, you'll need to do it in two places instead of one.
Release plugin instances from a role, and gather plugins into a role
These two operations complement one another.
From any plugin template, you can release the plugins from the role by clicking the "release plugin instances from template" link on the right side. No plugins are deleted, and no data is lost -- the plugins simply become individually managed, instead of being managed through the role.
You can immediately reverse this operation by navigating to the role, and clicking "gather plugins for this role." Gathering looks for identical individual plugins on all servers in the role, and allows you to consolidate those plugins under the roles so they can be managed together. Plugins can only be gathered if they have the same configuration, and the the same version of their code.
A common use case for releasing and gathering plugins: You have essentially the same plugin in multiple roles, and you want to manage them all together. You can release each plugin from its role, then re-gather them under a broader "umbrella" role. You can use this to undo the "Split plugin template into multiple roles" example outlined earlier.
What's the best way to handle multiple environments?
Say you have production and staging environments, and they are identical except MySQL needs a different password, the web heartbeat plugins need different IP addresses, and you don't want SMS notifications from staging.
The best approach is typically to create separate roles for each environment (for example: Application - Production, Application - Staging). Within each role, you can then configure the plugins appropriately. This will allow you to configure your staging environment independently from your production environment.
How do I override plugin configurations for just one or two plugin instances in a role?There are three options for this:
- Install as individual plugins: you can still install plugins individually on a server, outside of any role. If your role only has a few servers, you might be better off installing the plugins individually. If you already have the plugins installed via a role, you can use the "release plugin instances from role" feature outlined earlier.
- Use config lookups: If your plugins differ only via configurations, you can use configuration lookups to customize the configuration on each machine.
- Create separate roles: Use the "Split plugin template into multiple roles" feature to make separately configurable plugin templates, and configure the exceptional cases independently.
What's the best way to test out a plugin on a subset of app servers?
Say it's a Redis plugin. Create a new role, "Application - Redis". Add the Redis plugin to this role. Add the role to the subset of servers. You'll likely want to hide this role from your account homepage. If you eventually want to put Redis on all your servers, you can move the Redis plugin out of "Application - Redis" and into the main Application role.
One of our servers is having issues and sending out a bunch of alerts. Is there anyway to silence it?
No - the best workaround may be to just disable Scout on that server until the issues are sorted out.
Transitioning an existing account to roles
I have the "your account has been upgraded to roles" message. What do I do now?1) Create roles as needed
Click "Roles" in the top nav. Roles should match the functionality of your servers. Most accounts have a database role, an application role, etc. Populate your roles with servers, but don't add plugin templates just yet -- you'll probably want to organize your existing plugins into roles, per step (2).2) Organize your plugins
Since your account started out without roles, you'll need to take steps to organize your existing plugins into roles
Click onto a role, then click "gather plugins for this role." If there are similar plugins across all the servers in the role, you can add all those plugins to the role in one click. Once a plugin is assigned to a role, you can manage the plugin and its triggers in unison across all the role's servers.
You don't have to organize all your plugins into roles; you can leave some as individual plugins on a server. However, to fully take advantage of roles, you probably want to place as many plugins into roles as possible.3) Use your account key for new servers
Previously, you had to generate a unique key for each server in your account (or use cloud keys). Now, you have just one account-wide key, which you'll see on the "add server" page after conversion. You'll see instructions for the Chef recipe there as well.
I'm transitioning to roles from an old-style account. Is it safe for me to just switch out client keys w/the account key?
Yes. Scout fetches server by the account key and hostname, and since the hostname hasn't changed, the same plugins will be fetched.
I'm transitioning to roles from an old-style account. Do I need to change the Scout key on my existing servers?
You don't need to change anything. Your existing servers will continue to report data with their legacy keys. However, for new servers added to you're account, you'll need to use the new account key.
Inserting Report Data into Scout
Can I associate report data with an arbitrary time?
No - you can't insert historical data into Scout.
Scout isn't graphing fields it should be (or vice versa). Does report data need to fit a certain format?
If a field has a
nil value, it won't be graphed. Instead, return zero.
Is there a maximum size for a report field name?
Yes - the maximum size of a report field name is 30 characters. If the field name exceeds 30 characters, then it will be truncated to 30 characters. For example:
# use a report field that is longer than 30 characters field_name = 'This is a very long report field name' field_name.size => 37 # This becomes the following in Scout: field_name[0..29] => "This is a very long report fie"
How can I automatically remove instances from Scout when they are shutdown?
Create a shutdown script linked in the
/etc/rc0.d directory on the server. Scripts in this directory are
run on shutdown. An example script:
#!/usr/bin/env ruby require 'rubygems' require 'scout_api' scout = Scout::Account.new('ACCOUNT_NAME',EMAIL,PASS) server=Scout::Server.first(:host => Socket.gethostname) Scout::Server.delete(server.id)
The script requires the
Can I reduce the no data alerting threshold to less than 30 minutes?
In the past, this threshold was adjustable, but it frequently resulted in unnerving false positives at lower thresholds. These alerts were often used as an indicator of a server outage, but a breakdown in a server's ability to contact Scout doesn't necessarily mean a server is down.
We suggest using an external monitoring service like Pingdom to check if a server can't be pinged and/or isn't accepting SSH/TCP connections. Failed external checks and a lack of internal metrics from Scout often indicate that a server really is down.
Read more on our blog.
I'm having problems authenticating to retrieve RSS Feeds.
Scout's RSS Feeds are protected with Basic Authentication and use the same credentials as your Scout login (email & password). However, you can't type the '@' symbol of your email address in a URL. Instead, replace '@' with '%'. For example, if my email address was "email@example.com": http://dog%scoutapp.com:firstname.lastname@example.org/apple/activities.rss.
Will Scout start automatically when my server is rebooted?
We suggest running Scout regularly through Cron. This means you don't have to do anything special to ensure Scout runs when your server is restarted.
I have a number of servers I'd like to install Scout on. Are there any auto-deployment options?
Yes - see our Chef Recipe for deploying Scout to multiple servers.
How do I uninstall Scout?
Remove the Crontab line for Scout. To access your Crontab file: sudo vi /etc/crontab or crontab -e .
In addition, you can run sudo gem uninstall scout to remove the Scout gem from your system.
What happened to cloud keys?
Cloud keys were discontinued with the introduction of server roles. Server roles are a better, more flexible solution to the bulk server management problem. They also save accounts money by not using a server instance as a template.
What happened to plugin copy/paste?
The plugin copy/paste feature was discontinued with the introduction of server roles. Server roles are more powerful solution to the bulk server management problem.