Ignite Those Horrible Scripts

Published on:
Tags: script, shell, zsh

The past week was spent dealing with an infuriating data crunching task. Chris and I decided to celebrate by killing our poorly written Ruby cli scripts with fire. Printing the scripts and going out back to set them on fire would be more dramatic. Though it probably would be appreciated by the other businesses which we share space with.

I took a breather and looked into zsh functions to accomplish this. Here’s what I came up with:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
rageflip() { echo " ┻━┻ ︵ヽ(\`Д´)ノ︵ ┻━┻  $*" }

has_utility() { hash $1 2>/dev/null }

# Wrapper around `rm -rf` expressing callers loathing and disdain for the
# provided files.
#
# Inspiration was Homebrew's output:
# 🍺  /usr/local/Cellar/tmux/1.9: 15 files, 628K, built in 25 seconds
#
# For additional file stats, specific to code files, consider installing the
# Count Lines of Code (cloc) tool: http://cloc.sourceforge.net/
#
#   brew install cloc
#   sudo apt-get install cloc
#
ignite() {
  if (( $# == 0 )) then echo "USAGE: ignite file [file] ..."; return; fi
  local total_lines
  local human_size
  local lc_blank
  local lc_comment
  local lc_code
  echo "BURN IT ALL!!! $(rageflip)"
  for i do
    # If the file is empty we have 0 lines
    human_size=$(ls -lh $i | awk '{ print $5 }')
    total_lines=${$(sed -n '$=' $i):-0}
    stats="$total_lines lines"

    if has_utility cloc; then
      # Setup some local variables regarding file stats
      lc_blank=
      lc_comment=
      lc_code=
      eval $(
        cloc $i --quiet | tail -2 | head -1 |
        awk '{ print "lc_blank="$3, "lc_comment="$4, "lc_code="$5 }'
      )
      if [ ! -z $lc_blank ]; then
        stats="$lc_code loc, $lc_comment comments, $lc_blank whitespace lines"
      fi
    fi

    rm -rf $i
    echo "🔥  $i: $stats, $human_size"
  done
}

Now we can sit back and watch it all burn:

1
2
3
4
5
6
$ ignite script/crunch-*
BURN IT ALL!!!  ┻━┻ ︵ヽ(`Д´)ノ︵ ┻━┻
🔥  script/crunch-dump-sessions: 5 loc, 5 comments, 2 whitespace lines, 330B
🔥  script/crunch-normalize: 47 loc, 14 comments, 11 whitespace lines, 2.2K
🔥  script/crunch-import-csv: 101 loc, 2 comments, 20 whitespace lines, 2.7K
🔥  script/crunch-timestamp: 203 loc, 44 comments, 38 whitespace lines, 8.4K

Though at this point, it might as well be a full script file. If I do that, I might as well just write it as a Ruby cli…

Pitfalls of Testing Scripts With Load - Part 2

Published on:

Check out the full Testing Scripts with Load series: part 1, part 2, part 3

Scope Smashing

Take the following script and spec:

1
2
3
4
5
#!/usr/bin/env ruby

def current_user
  'Bob'
end
1
2
3
4
5
6
7
8
9
10
11
12
13
require 'spec_helper'

def current_user
  'Alice'
end

describe 'Running the server locally' do
  it 'logs that it is running' do
    load 'script/current-user-smash'

    expect(current_user).to eq 'Alice'
  end
end

Alas, the spec fails with:

1
2
expected: "Alice"
     got: "Bob"

This is due to how load works:

If the optional wrap parameter is true, the loaded script will be executed under an anonymous module, protecting the calling program’s global namespace. In no circumstance will any local variables in the loaded file be propagated to the loading environment.

While it is easy to spot the issue this time, that’s not normally the case. Say if the additional method is define by a gem or in supporting file. Or if you are testing multiple scripts that each define the same top-level methods. These conditions will result in very strange and difficult to debug failures. Of course, it’s always a good idea to not define top-level methods to begin with.

Instead always pass the additional wrap parameter. Here I’ve named it as a descriptive inline variable to reveal it’s intent:

1
2
3
4
5
it 'logs that it is running' do
  load 'script/current-user-smash', _in_sandbox = true

  expect(current_user).to eq 'Alice'
end

Discussion About Open Classes and Monkey Patching at #retroruby

Published on:
Tags: retroruby, ruby

While at RetroRuby 2014 a very good question was asked in the newbie track. The following code sample had just been shown on some basic awesomeness of Ruby:

1
2
"Words" * 2
# => "WordsWords"

One of the participants then asked why that worked but the inverse didn’t:

1
2
2 * "Words"
# TypeError: String can't be coerced into Fixnum

Oops what just happened?

In Ruby, virtually everything is accomplished by sending a message to another object. Above we would say:

Send the message * to "Words" with parameter 2

Send the message * to 2 with parameter "Words"

“Sending a message” is the Java equivalent of calling a method. Another way to write the above is:

1
2
"Words".*(2)
# => "WordsWords"

Here we used the normal “method” calling syntax: obj.method(args)

You’ll also probably see the following:

1
2
"Words".send(:*, 2)
# => "WordsWords"

This time we explicitly sent the message: obj.send(message, args)

With send Ruby doesn’t check if the message you passed was supposed to be public or private. Generally, what you wanted to do was dynamically send the message while still making sure to respect the public API of the object. To do this you should use public_send instead: obj.public_send(message, args).

So back to the original issue. Both String and Fixnum respond to the message *.

However, String’s implementation knows what to do when the argument is a Fixnum. When we reverse it, the Fixnum implementation doesn’t understand what to do with a String argument.

How to fix this?

Well you probably shouldn’t. But just for fun we’ll use Ruby’s open class behavior to monkey patch Fixnum’s * implementation.

1
2
3
4
5
6
7
8
9
10
11
class Fixnum # We just re-opened the class

  def *(arg) # We're redefining the * message - this destroys the
             # previous implementation!!!
    arg * self
  end

end

2 * "Words"
# => WordsWords

It worked!! Before you go doing this to everything, be aware we’ve now lost the ability to do normal multiplication. Additionally, we’ll overflow the stack trying to multiply two Fixnums.

1
2
2 * 2
# SystemStackError: stack level too deep

Wrap Up

In Ruby, most things are messages sent from one object to another. Some times the language gives a little extra syntactic sugar to make sending the message more readable.

Additionally, open classes and monkey patching have their uses. Just be aware that with great power comes great responsibility. Always stop to ask yourself if this is really a good idea; and try to never change existing behavior if you can.

Level Up!

So what if we just wanted to extend the existing functionality. There are a lot of ways to do this and covering each, along with the pros and cons is very out of scope for this post. I’m just going to demonstrate one way to illustrate how it could be done.

You’ll need a new irb or Pry session so we get the original implementation for Fixnum back.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
class Fixnum # We just re-opened the class

  # We need to store a reference to the original implementation One way to do
  # this is to create an alias for the original implementation so we can call
  # it later. Be sure to pick a descriptive name so that it isn't accidentally
  # overwritten by something else.
  alias :star_without_string_support :*

  def *(arg) # Redefine the message destroying the previous implementation!!!
    if arg.is_a? String
      arg * self
    else
      star_without_string_support arg  # use the original implementation
    end
  end

end

Live-ing Dangerously With Rails

Published on:
Tags: rails, ruby

Recently at work, we’ve been spinning up a bunch of demo apps on Heroku. This has worked out well for us so far. However, some of these apps require background data crunching. This hasn’t been a problem as we just add a Rake task, or two, and set the Heroku scheduler to take care of things.

Then comes the day when the sales team says:

“Oh hey. Can we re-run all the numbers but with different configurations?”

Sure, that’s not a problem you think:

1
$ heroku run rake redo_all_the_things[config.yml]

However, there’s a few issues with this:

  1. The config.yml needs to be on Heroku
  2. This counts against your active dyno time

See the thing is, Heroku doesn’t let you create files that aren’t already in your VC system from outside the app. It’s now sorta possible on Cedar, but not really for this purpose.

Additionally, if these tasks turn out to be long running, that’ll quickly eat up the overhead on the ‘free’ dyno. For demo apps of non-paying customers that’s not always an option.

Enter the live environment.

Just add a new live environment to the Rails application. This allows us to run Rake locally, but against the production database. Additionally, allowing for custom configuration files to just be stored on our local system. And we don’t spin up a new dyno on Heroku. Win, win!

To set this up:

  • Update the Gemfile
1
2
3
4
5
# Gemfile

# We use dotenv-rails to help manage environment configs locally.
# Add our new environment:
gem 'dotenv-rails', groups: [:development, :test, :live]
  • Add the live specific sample file: .env.live-example
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
# .env.live-example

# Rename this file to `.env.live` and it will be automatically sourced by
# rails. Uncomment the settings for them to be picked up.
#
# Grab the settings from the output of:
#
#  $ heroku config
#
# Running against the Heroku production DB directly
# BE CAREFUL!! YOU HAVE BEEN WARNED!!
#DATABASE_NAME=WARNING
#DATABASE_HOST=DANGER
#DATABASE_PORT=WILL
#DATABASE_USER=ROBINSON
#DATABASE_PASSWORD=SERIOUSLY
  • Update the database.yml file
1
2
3
4
5
6
7
8
9
10
# config/database.yml

live:
  adapter: postgresql
  encoding: unicode
  database: <%= ENV['DATABASE_NAME'] %>
  host: <%= ENV['DATABASE_HOST'] %>
  port: <%= ENV['DATABASE_PORT'] %>
  username: <%= ENV['DATABASE_USER'] %>
  password: <%= ENV['DATABASE_PASSWORD'] %>

Now running our tasks is as simple as:

1
$ RAILS_ENV=live rake redo_all_the_things[config-fun-edition.yml]

Pitfalls of Testing Scripts With Load - Part 1

Published on:

Check out the full Testing Scripts with Load series: part 1, part 2, part 3

Previously it was shown how to use load to test scripts. As with all techniques, there are some drawbacks to using load.

Hidden Require Dependency

Take script:

1
2
3
4
5
6
7
8
#!/usr/bin/env ruby
$:.unshift File.join File.dirname(__FILE__), "..", "lib"

require 'local_server'

logger = Logger.new($stderr)

LocalServer.new(logger).run

and passing spec:

1
2
3
4
5
6
7
8
9
10
11
12
13
require 'spec_helper'
require 'logger'
require 'stringio'

describe 'Running the server locally' do
  it 'logs that it is running' do
    io = StringIO.new
    allow(Logger).to receive(:new).and_return(Logger.new io)

    expect{ load 'script/local-server' }.to change{ io.string }
      .to include 'SERVER: starting up'
  end
end

However, when the script is run standalone, it errors with:

1
uninitialized constant Logger (NameError)

Be aware that since load happens in the current spec context, a missing require may not be noticed if it is required by the spec.

Whenever possible have at least one test that shells out as a sanity check.