Getting More from Skylight

Learn how Skylight works. Enable instrumentation for additional libraries and add custom instrumentation to your own code.

Available Instrumentation Options

Rails

See the Rails setup instructions.

In Rails applications, we use ActiveSupport::Notifications to track the following.

• Controller Actions
• Controller File Sending
• View Collection Rendering
• View Partial Rendering
• View Template Rendering
• View Layout Rendering
• ActiveRecord SQL Queries
• ActiveRecord Instantiation
• ActiveSupport Cache
• ActiveStorage

Grape

See the Grape setup instructions.

• Endpoint Execution
• Endpoint Filters
• Endpoint Rendering

Sinatra

See the Sinatra setup instructions.

• Endpoint Execution

Background Jobs

See the Background jobs setup instructions

Active Job Enqueueing

Enabled by default

• Enqueuing of Active Jobs

ActiveModel::Serializers

Add active_model_serializers to probes list.

• Serialization

Coach

Enabled by default

• Middleware

Couch Potato

Add couch_potato to probes list.

• Database Queries

Elasticsearch

Add elasticsearch to probes list.

• Search Queries

Excon

Add excon to probes list.

• HTTP Requests

Faraday

Add faraday to probes list.

• HTTP Requests

Graphiti

Enabled by default for Graphiti 1.2+

• Resource Resolving
• Resource Rendering

GraphQL

Add graphql to probes list

• Available in Skylight version 4.2.0 and graphql-ruby versions >= 1.7.
• Traces invocations of GraphQL::Schema#execute and GraphQL::Schema#multiplex

The GraphQL probe conditionally adds the GraphQL::Tracing::NotificationsTracing module to your schema the first time a query is executed (You may see a note about this in STDOUT).

Using Named Queries

In order for Skylight’s trace aggregation to work properly, we highly encourage the use of named queries.

Skylight names your endpoint based on all the queries sent in a single request. This means that all similar queries should be grouped together. Additionally, if you send two named queries together (a multiplexed query), your endpoint name will be a combination of those two query names. You can learn more about how this works in our blog post.

Anonymous queries will also be tracked, but instrumentation will be disabled below the Schema#execute span, as we don’t believe aggregating divergent anonymous queries will provide you with actionable insights.

HTTPClient

Add httpclient to probes list.

• HTTP Requests

Middleware

Enabled by default

• Tracks Rack Middleware in Rails applications.

Mongo (Official Driver)

Add mongo to probes list.

• Database Queries

Moped

Add moped to probes list.

• Database Queries

Mongoid

Add mongoid to probes list.

Depending on the version, Mongoid either uses Moped or the official Mongo driver under the hood. This probe will just enable the correct probe for one of these two libraries.

Net::HTTP

Enabled by default

• HTTP Requests

Redis

Add redis to probes list.

• All Commands

Sequel

Enabled automatically with Sinatra or Add sequel to probes list.

• SQL Queries

Tilt

Enabled automatically with Sinatra or Add tilt to probes list.

• Template Rendering

Custom App Instrumentation

The easiest way to add custom instrumentation to your application is by specifying methods to instrument, but it is also possible to instrument specific blocks of code. This way, you can see where particular methods or blocks of code are called in the Event Sequence. Check out an example of custom instrumentation in action. Be sure to follow our custom instrumentation best practices in order to avoid causing errors in your Skylight trace.

Method instrumentation

Instrumenting a specific method will cause an event to be created every time that method is called. The event will be inserted at the appropriate place in the Event Sequence. You can see an example of how this looks below.

To instrument a method, the first thing to do is include Skylight::Helpers into the class that you will be instrumenting. Then, annotate each method that you wish to instrument with instrument_method.

class MyClass
  include Skylight::Helpers

  instrument_method
  def my_method
    do_expensive_stuff
  end
end

You may also declare the methods to instrument at any time by passing the name of the method as the first argument to instrument_method.

class MyClass
  include Skylight::Helpers

  def my_method
    do_expensive_stuff
  end

  instrument_method :my_method
end

By default, the event will be titled using the name of the class and the method. The event name in our previous example would be: MyClass#my_method. Alternatively, you can customize the title by using the :title option.

class MyClass
  include Skylight::Helpers

  instrument_method title: 'Expensive work'
  def my_method
    do_expensive_stuff
  end
end

Block instrumentation

Method instrumentation is preferred, but if more fine-grained instrumentation is required, you may use the block instrumenter.

class MyClass
  def my_method
    Skylight.instrument title: "Doin' stuff" do
      step_one
      step_two
    end
    step_three
  end
end

Just like above, the title of the event can be configured with the :title option. If you don’t add a title, the default for block instrumentation is app.block. We recommend creating your own titles to differentiate the various instrumented code blocks from one another in the Skylight UI. If you don’t, they will be aggregated together, which won’t be helpful for you.

Custom Instrumentation Best Practices

Use Static String Literals for Custom Titles

Use Method Instrumentation Wherever Possible

While block instrumentation can be very useful for zeroing in on problematic minutiae, it can lead to overly complex request traces. Prefer method instrumentation where possible.

Avoid Over-Instrumenting Your Action

Add custom instrumentation only where necessary to understand an issue in order to avoid overwhelming yourself with too much information. Additionally, avoid instrumenting a method or block that happens a ton of times in repetition (e.g. a method called within a loop). Doing so may cause your event to exceed the size limit for an event trace in the Skylight agent.

Custom instrumentation in action

When we first implemented GitHub sign-in and sign-up, we noticed the act of signing in or signing up with GitHub was pretty slow. We wanted to see which methods in our GitHub integration were taking the longest, so we added a bit of method instrumentation. The Event Sequence for the endpoint that included those methods then looked more like this:

Clearly the OctokitService#repo_summaries method was the culprit of the slowdown, so we knew where more refactoring was needed and that we might even consider creating a worker to take over much of this process (just look at how long those GitHub API requests are!). Custom instrumentation to the rescue!

Muting Skylight instrumentation

While we do our best to collect actionable data, we can occasionally instrument too much, which may result in a trace exceeding its predefined limits. For example, if you’ve seen the E0003 error, it may be due to over-instrumentation. In some cases, you may want to dig deeper than these limits allow, so we’ve introduced Skylight.mute. You might use this if:

• You have too much data (as described above), and would like to dig deeper than our data limits allow
• There is a particularly volatile section of code that does not aggregate well (for example, a webhook endpoint that completely changes behavior based on request paramters)
• You have a sensitive area of the application that should not be instrumented due to legal or other non-technical reasons.

Skylight.mute is available in version 4.2.0 and higher.

Ignoring blocks of code

Skylight.mute do
  Skylight.instrument(title: "I-wont-be-traced") do
    ...
  end
end

Ignoring portions of a request

Since mute is block-based, we’ve also added an unmute, which counteracts the effects within a mute block. If you would like to skip instrumentation for a portion of your request in favor of instrumenting a later method call, you may nest unmute inside the mute block:

def show
  Skylight.mute do
    some_expensive_method
    another_expensive_method
  end
end

def some_expensive_method
  # will _not_ be traced
  ...
end

def another_expensive_method
  Skylight.unmute do
    # code inside here _will_ be traced
    ...
  end
end

How Skylight Works

If you’re curious about how our instrumentation works, you’re in the right place.

Featherweight Agent

The Skylight agent is written primarily in Rust, a systems programming language on par with C. This means we have a very low overhead so it’s safe to run Skylight in production, even in memory limited environments like Heroku. You can read more about this aspect of our agent in our Featherweight Agent blog post and more about the agent’s 1.0 release in our announcment post.

Normalizers

Our preferred method of instrumentation is ActiveSupport::Notifications events. When a library has added instrumentation, all we need to do is subscribe to the event and do a little bit of normalization of the data.

To standardize this process, we introduced Normalizers. Each type of instrumentation has its own normalizer which handles the, well, normalization. You can take a look at some of them in the source.

Probes

While we think most libraries should include ActiveSupport::Notifications (anyone can subscribe to these notifications, not just Skylight), unfortunately, many still don’t. In these circumstances, we have to carefully monkey-patch the libraries at their key points.

To make sure we do this in a sane fashion, we developed Probes. Probes are small classes that keep an eye out for specific modules and then hook into them when they’re loaded. All probes can be disabled in the event of any conflicts and we only autoload probes that we have a high degree of confidence in. You can take a look at some probes in the source.

And, since we don’t really like having to monkey-patch things either, when at all possible, we submit pull requests to relevant projects to add in ActiveSupport::Notifications.

For more information about how to add an existing probe (such as Mongo, Moped, Excon, etc.) to your Skylight setup, see the Advanced Setup Probes page.