Tag: Errbit

Errbit + HTTPS: Setting up Errbit reporter (Airbrake v5 gem) to work with self-signed HTTPS certificate

Note: If you're using old Errbit version (0.2.0, 0.4.0) and an old Airbrake version (v4) please refer to this manual to make it work with self-signed certificates.

Having an error catcher like Errbit behind SSL is generally a good idea. Especially when Errbit is hosted on a different server than you application (for example when you manage multiple apps with one Errbit instance). In many cases you will have a self-signed certificate (why would you pay for a cert for internal tool). If you try to use it with Airbrake, you will see following error:

Airbrake:
  HTTP error: SSL_connect returned=1 errno=0 state=unknown state: certificate verify failed

Unfortunately, global SSL certificates verification disabling hack (solution that used to work with Airbrake notifier v4) won't work:

# No longer working!
OpenSSL::SSL::VERIFY_PEER = OpenSSL::SSL::VERIFY_NONE

Luckily, Airbrake notifier is written pretty well, so hacking it (and disabling per request SSL certificate verification) is not hard at all. Here's a full code you need to place in config/initializers/errbit.rb to make it work:

module Patches
  module Airbrake
    module SyncSender
      def build_https(uri)
        super.tap do |req|
          req.verify_mode = OpenSSL::SSL::VERIFY_NONE
        end
      end
    end
  end
end

Airbrake::SyncSender.prepend(::Patches::Airbrake::SyncSender)

After that (and configuring Airbrake notifier), you can test it out like this:

Airbrake.notify :test

Making Errbit work faster by keeping it clean and tidy

Errbit is a great tool for collecting and managing errors from ruby applications. It's like Airbrake but can be self-hosted, so you can use it for intranet applications or any apps that should not send data to an external servers.

Errbit is a really good piece of software, unfortunately it can get pretty slow, when you use it extensively. Errbit gets slower and slower mostly because of number of problems that are getting stored in the DB. Even those that are resolved aren't removed by default from it, so after some time, the database can get really huge. It can be an issue especially when you have multiple apps connected with one Errbit instance and they report errors in a huge quantity.

There are two easy steps that you should take to prevent this from happening:

  1.  Remove all resolved issues from the DB (not just hide them)
  2. Auto-resolve issues that are older than 2 weeks and that don't occur anymore

Both this tasks should be executed periodically, so we will use crontab to achieve this.

Removing all resolved issues

There is a rake task for that built in Errbit already. To execute it, just run following command:

bundle exec rake errbit:clear_resolved

If you have Errbit from a long time, runnning this task can take a while. To add it to crontab, just execute crontab -e and paste following command:

0,30 * * * * /bin/bash -l -c 'cd /errbit_location && RAILS_ENV=production nice -n 19  bundle exec rake errbit:clear_resolved'

If you're interested why we use nice to exec this task, you can read about it here: Ruby & Rails: Making sure rake task won’t slow the site dow.

This cron task will be executed every 30 minutes and will automatically remove any resolved issues.

Auto-resolving issues that are older than 2 weeks and that don't occur anymore

It happens quite often, that one fix resolves more than one issue. Sometimes you might not even realise, that your fix, fixed multiple issues. How to handle such a case? Well lets just resolve any issues that didn't occur for at least 2 weeks. Unfrotunately there's no predefine Errbit rake task for this, so we need to write our own. To do this open lib/tasks/errbit/database.rake file and add following task:

desc 'Resolves problems that didnt occur for 2 weeks'
task :cleanup => :environment do
  offset = 2.weeks.ago
  Problem.where(:updated_at.lt => offset).map(&:resolve!)
  Notice.where(:updated_at.lt => offset).destroy_all
end

That way we will get rid of old, not resolved problems. This task should be also executed using crontab:

15,45 * * * * /bin/bash -l -c 'cd /errbit_location && RAILS_ENV=production nice -n 19  bundle exec rake errbit:cleanup'

Notice that we run it 15 minutes before the previous rake task, so resolved issues will be removed by the errbit:clear_resolved task.

Removing errors when heavy crashes occur

Warning! Use this code wisely. This will remove unresolved errors as well!

If you have systems that sometimes tend to throw (due to one issue) 100k+ errors that aren't getting squashed, you might think of using code presented below. It will automatically remove all errors if there are more than 2k of them. That way you won't end up having DB with too many errors.

desc 'Removes issues if we have more than given threshold'
task :optimize => :environment do
  treshold = 2000
  batch_size = 100

  App.all.each do |app|
    next unless app.problems.count > treshold

    batch = []
    app.problems.offset(treshold).each do |problem|
      batch << problem

      if batch.length == batch_size
        next if app.reload.problems.count < treshold

        Err.where(:problem_id.in => batch.map(&:id)).delete_all
        batch.each(&:delete)
        batch = []
      end
    end
  end
end

Of course you need to add it to crontab as well:

25,55 * * * * /bin/bash -l -c 'cd /errbit_location && RAILS_ENV=production nice -n 19  bundle exec rake errbit:optimize'

Copyright © 2024 Closer to Code

Theme by Anders NorenUp ↑