This site is built using the famous web technologies Ruby, Nokogiri, HAML and jQuery. Basically I do not own a Rails enabled webspace, so I decided that I just crawl the data on my local Mac, put the data into some HAML templates and generate some HTML, which gets a bit of dynamic using jQuery with its great DOM manipulation functionality.

Some words about the crawling. Basically I’m using the standard Ruby Net::HTTP classes to make simple get request to the official LoL website to gather information about which champions and items are available. After collecting the basic data I fetch all patch notes with simple HTTP GET request.

The so acquired HTML pages are parsed using the famous Nokogiri parser. So all the data comes together – champions, items, changes.

First I started off creating some custom HTML. This turned out – as always – to be too inflexible as the app grew. So I started to include HAML and built some HAML templates, which basically are saved as HTML afterwards.

Having the feeling that the site needs some search or filter function I first thought that I need some PHP on the server side. But I ended up using jQuery to filter the champions and items, which really turned out to be easy, using the built in jQuery $.each and some regular expressions.

To conclude this post I must say that all this technologies are great and I can only recommend to try them out and use them! Drop me a comment if you like a more detailed post about this topics.

1
2
3
4
5
6
7
8
9
10
11

$ = jQuery
 
methods =
  init: (options) ->
    obj = @
    # do something
 
$.fn.your_plugin_name = (method) ->
  if method of methods
    return methods[ method ].apply( this, Array.prototype.slice.call( arguments, 1 ) );
  else $.error( "Method #{method} does not exist on jQuery.your_plugin_name" );

This coffeescript compiles to:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

var $, methods;
$ = jQuery;
methods = {
  init: function(options) {
    var obj;
    return obj = this;
  }
};
$.fn.your_plugin_name = function(method) {
  if (method in methods) {
    return methods[method].apply(this, Array.prototype.slice.call(arguments, 1));
  } else {
    return $.error("Method " + method + " does not exist on jQuery.your_plugin_name");
  }
};

You would call your plugin e.g. this way (from javascript):

$('#target_div').your_plugin_name('init', {your: "option"});

If you are still waiting for the *Yeah, I have to use coffeescript!”-moment read this:

The lines of code of my jQuery plugin were reduced by 50% and the code is much more readable now! Don’t miss the coffeescript page which will give you much more arguments for coffeescript as I could ever do here.

1
2
3
4
5
6
7

f = open("|gem list")
output = f.read()
 
output.each_line do |l|
	gem_name = l.split(" ").first
	system "gem uninstall --a --ignore-dependencies #{gem_name}"
end

Basically this script just executes the “gem list” command, fetches the output, extracts the gem name and uninstalls it.

Environment:

• Mac Os X – Leopard 10.6.6
• MacPorts 1.9.2

Start of the installation

$ sudo port selfupdate
$ sudo port install git-core

1st Problem

Error: Target org.macports.build returned: shell command failed (see log for details)
Error: Failed to install perl5.12

Perl5 is a compilation dependency, but somehow not declared, so install it manually:

$ sudo port install perl5

And again:

$ sudo port install git-core

2nd Problem

Error: db46 requires the Java for Mac OS X development headers.

https://connect.apple.com/cgi-bin/WebObjects/MemberSite.woa/wa/getSoftware?bundleID=20719

Interested in why this error comes up?

http://comments.gmane.org/gmane.os.apple.macports.user/22266

Apple’s most recent security update prior to Mac OS X 10.6.5 (presumably included in 10.6.5 also) deletes a
file that was previously part of the OS, and is now only part of that development package. We don’t know why
Apple made this change, only that db46 and a handful of other ports break without it, so we detect this and
advise you how to fix it.

Visit with the Apple page and fetch the development headers – be aware that you need an Apple-ID.

And again:

$ sudo port install git-core

Success!

Basically you can define a lot of tasks, which can be executed, may have prerequisites and can do a lot boring tasks, from file operations to building gems.

One great thing is that you can easily deploy your whole project to your FTP server. Searching around I did not find a lot of information about that – this is the reason I’ll write about it.

Rake::FtpUploader
Rake has a built in class for handling simple FTP uploads, but this class is not well documented. You can use it this way:

1
2
3
4
5
6
7
8
9

require 'rake'
require 'rake/contrib/ftptools'
 
task :upload_all do
  Rake::FtpUploader.connect('/path/on/server', 'your_host', 'your_user', 'your_pw') do |ftp|
    ftp.verbose = true # gives you some output
    ftp.upload_files("./your/favorite/folder/**/*")
  end
end

The ftp.upload_files method takes a wildcard parameter, which is the same as you would feed into the Dir[wildcard] class. At the end, the FtpTools just use Dir to find all files. For the wildcard it is good to know, that a single * says “upload all files and folders”, but a **/* says “upload all files and folders in a recursive way, including subfolders and their files. Thus said, “./**/*” would upload everything from the base dir of your rakefile.

Extending Rake’s FtpTools for deletion
In my case I was using rake to upload files for a PHP project – written with the symfony framework. Symfony – as many other frameworks – has a cache which needs to be cleared if some settings for the view change. As the cache is simply a folder on the FTP server, I thought that rake could be used to easily clear the cache. Reminding myself of the fact that everything in Ruby is just an object, I just extended the FtpUploader class:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45

module Rake
  class FtpUploader
 
    # Deletes all files in a folder like cache/management. Does
    # not delete folders
    def delete_files(folder)
      delete_files_recursive(folder)
    end
 
    private
 
    # starts with a folder, like cache/management and deletes
    # all files recursively BUT not the folders.
    def delete_files_recursive(file_or_folder)
      folder = true
      begin
        @ftp.chdir(file_or_folder)
      rescue
        # this is a file, no folder => delete
        folder = false
      end
 
      if folder
        file_list = @ftp.nlst('*')
        file_list.each { |f| delete_files_recursive(f) }
        @ftp.chdir('..')
      else
        puts "Delete #{file_or_folder}" if @verbose
        delete file_or_folder
      end
 
    end
 
    # deletes a single file
    def delete(file)
      puts "Deleting #{file}" if @verbose
 
      begin
        @ftp.delete(file)
      rescue
        puts "Could not delete file #{file}"
      end
    end
  end
end

This snippet only deletes files, no folders. In my case this is wanted, as I’m not sure what the cache does if you destroy the folder structure, but nevertheless it could be easily extended. After all files of a folder are deleted, after the @ftp.chdir(‘..’) is executed, the folder could be deleted.
Usage

1
2
3
4
5
6
7

namespace :cache do
  task :clean
    Rake::FtpUploader.connect('/path/on/server', 'your_host', 'your_user', 'your_pw') do |ftp|
      ftp.delete_files('cache/subfolder')
    end
  end
end

Often I found myself checking the comments for my game iBox3D using http://www.appcomments.com. But the problem is that you can only display the comments by country and my app does so far only have comments in Germany, USA and UK. But the website does not show where you have comments or not. It was hard to say, if there were new comments or not. So I decided to write a small ruby script which should compile all comments and display them. The script evolved as I continued to optimize it:

• v1: I used WATIR to automate the internet explorer and to get the HTML from the website
• v2: I used the built in RSS lib to fetch the RSS
• v3: I used simple-rss to fetch the RSS
• v4: I used simple-rss to fetch the RSS and used threads to do that for all countries simultaneously

This post will handle following topics:

• Ruby 1.9
• Watir
• RSS with the build-in RSS lib and simple-rss
• utf8 vs. ASCII
• Threading
• JRuby and Ruby

Version 1: Watir

Watir is a lib that helps to automate the browser. For those who want to install Watir at Ruby 1.9 you need to install the ruby devtools and then install the gem with the –platform=ruby option. Here is the script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68

require 'watir'
 
### hide or show the internet explorer
$HIDE_IE = true
 
# declare some variables
source_url = "http://appcomments.com/app/iBox3D"
comments = Hash.new
country_links = Array.new
 
### open the internet explorer and navigate to the source url
ie = Watir::IE.new()
ie.goto(source_url)
 
### fetch the links to the different countries
ie.links.each do |link|
	if link.href =~ /iBox3D\?country=(\d*)$/
		country_links < < link.href
	end
end
 
### loop over all country links
country_links.each do |link|
	# go to the country specific link
	ie.goto(link)
 
	# due to unknown reasons sometimes the DIV was not there => reload
	until  ie.div(:id => "review_dropdowns").exists? do
		sleep(1)
		ie.goto(link)
		sleep(5)
	end
 
	# get the DIV with id = review_dropdowns
	review = ie.div(:id => "review_dropdowns")
	# get the country text from the SPAN in the DIV
	country = review.divs.first.spans.first.text
	# loop over all DIVs
	ie.divs.each do |d|
		# check the DIV class, if it has "comment", then proceed
		if d.class_name.downcase.match(/\bcomment\b/i)
			# initialize a hash for the comment information
			inf = Hash.new
			# the first link in the div is the title/header
			inf[:header] = d.links[1].text
			# the second link in the div is the user
			inf[:user] = d.links[2].text				
			# loop over all DIVs in the DIV and find the comment_right class (rating)
			d.divs.each do |star_div|
				if star_div.class_name.downcase.match(/\bcomment_right\b/i)
					# count the start image tags
					counter = 0
					star_div.images.each do |img|
						counter += 1
					end
					inf[:stars] = counter
				end
			end
			# get the description of the rating
			inf[:text] = d.ps.first.text
			# store the information in the comments hash
			comments[country] = inf
		end
	end
end
 
### show the result
puts comments.inspect

To understand the coding I suggest that you use Firebug or the Internet Explorer Developer Tools to have a look at the HTML of the coding of http://appcomments.com/app/iBox3D. The comments should be enough to understand the coding – at least I hope so. Nevertheless I want to tell you why I did not stop here and continued to search for alternatives:

• if the html structure changes, the script will not work any more (no interface contract)
• it takes very long to start of the internet explorer and to visit each page
• this script is not platform independent, for other platforms you would have to use e.g. firewatir
• if I would use threads for parallelisation it would be very memory intense, because many IE instances take a lot memory

Then I noticed the RSS feed on appcomments.com.

Version 2: Ruby and RSS

RSS in ruby should be simple:

1
2
3
4
5
6
7
8
9
10

require 'rss/1.0'
require 'rss/2.0'
require 'open-uri'
 
source = 'http://appcomments.com/rss/376860218?country=143443'
 
content = "" # raw content of rss feed will be loaded here
open(source, :proxy => "http://proxy:8080") do |s| content = s.read end
rss = RSS::Parser.parse(content, false)
[...]

But try to run it! It will fail with the error:


C:/dev/runtime/Ruby191/lib/ruby/1.9.1/rss/rexmlparser.rb:24:in `rescue in _parse': This is not well formed XML (RSS::NotWellFormedError)
#encoding ::CompatibilityError: incompatible encoding regexp match (UTF-8 regexp with ASCII-8BIT string)
[...]
Exception parsing
Line: 12
Position: 462
Last 80 unconsumed characters:
< ![CDATA[Brilliantes Spiel fürs iPhone]]>
[...]


German umlauts! (ä,ö,ü) Somehow ruby 1.9, which claims to have UTF-8 support, introduces a lot problems. The german character ü cannot be parsed by the internal rexml. I found the simple solution to force ruby to think that the content string is utf-8, using the force_encoding method. Then parsing works.

1
2
3
4
5
6
7
8
9
10

require 'rss/1.0'
require 'rss/2.0'
require 'open-uri'
 
source = 'http://appcomments.com/rss/376860218?country=143443'
 
content = "" # raw content of rss feed will be loaded here
open(source, :proxy => "http://proxy:8080") do |s| content = s.read end
content.force_encoding('utf-8')
rss = RSS::Parser.parse(content, false)

Version 3: Simple-RSS

Before I found that out I tried the ruby lib simple-rss, which could parse the ü (german umlaut). Nevertheless I had to do the same trick as above when I wanted to access the parsed content. At this point I want to introduce the next evolution step of my script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

require 'rubygems'
require 'simple-rss'
require 'open-uri'
 
$base_html = < <EOF
<?xml version="1.0" encoding="utf-8"?>
< !DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>@@title</title>
</head>
<body>
EOF
 
$base_html_2 = < <EOF
</body>
</body></html>
EOF
 
def get_content(url)
	content = String.new
	open(url, :proxy => "http://proxy:8080") do |s| 
		content = s.read end
	return content
end
 
def utf8(string)
	return string.force_encoding('utf-8')
end
 
source = "http://appcomments.com/app/iBox3D"
countries = get_content(source).scan(/a href='\?country=(\d*)'.+?>(.+?) "http://proxy:8080")
 
	next if rss.items.size == 0
	html < < "<hr/><h1>#{country[1]}</h1>" 
	rss.items.each do |i|
		html < < "<div><h3>#{utf8 i.title}</h3><p>#{utf8 i.description}</p>"
	end
end
 
### write everything to a file
html < < $base_html_2
local_filename = "appcomments.html"
File.open(local_filename, 'w:utf-8') do |f|
	f.write(html)
end

Besides the advantages listet below I also added a file output to an HTML file. This is because the RSS description tag contains HTML which can be easily dropped to a HTML file.

Advantages:

• RSS is a standardized protocoll, so the structure won’t change in future (interface contract)
• instead of opening the browser to perform the scraping open-uri is used, which is faster and not so memory consuming
• this script should run on many platforms, including linux, mac os x and windows
• the process of making the web request is way easier

Version 4: Threading

Still my script took very long. No wonder, it had to make a GET request per country and one additional for the overview site to get all the country codes. But using ruby built-in threads it is easy to make all requests parallel! This speeds up the whole script:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76

require 'rubygems'
require 'simple-rss'
require 'open-uri'
 
### define the HTML basis
$base_html = < <EOF
<?xml version="1.0" encoding="utf-8"?>
< !DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
    "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>@@title</title>
</head>
<body>
EOF
 
$base_html_2 = < <EOF
</body>
</body></html>
EOF
 
### get request to a URL
def get_content(url)
	content = String.new
	open(url, :proxy => "http://proxy:8080") do |s| 
		content = s.read end
	return content
end
 
### forces the encoding in UTF-8
def utf8(string)
	RUBY_PLATFORM == 'java' ? string : string.force_encoding('utf-8')
end
 
### define the basic URLs and the filename
source_url = "http://appcomments.com/app/iBox3D"
rss_url = "http://appcomments.com/rss/376860218?country="
local_filename = "appcomments.html"
 
### get the main page to get all countries
countries = get_content(source_url).scan(/a href='\?country=(\d*)'.+?>(.+?)
threads = []
countries.each do |country|	
	threads < < Thread.new do
		# construct the URL with the number of the country
		url = rss_url + country[0]
		# get the RSS feed from the URL
		rss = SimpleRSS.parse open(url, :proxy => "http://proxy:8080")
		# construct the HTML with the country information and the review
		country_html = String.new		
		# go to the next country if there are no reviews
		next if rss.items.size == 0
		# construct the country header
		country_html < < "<hr/><h1>#{country[1]}</h1>" 
		# construct a div for each review
		rss.items.each do |i|
			country_html < < "<div><h3>#{utf8 i.title}</h3><p>#{utf8 i.description}</p>"
		end
		# set the thread variable for later access
		Thread.current["html"] = country_html
	end
end
 
### join the threads and construct the HTML
threads.each do |t|
	# join the threads
	t.join
	# construct one big HTML chunk out of the small HTML junks
	html < < t["html"] unless t["html"].nil?
end
 
### write everything to a file
html << $base_html_2
File.open(local_filename, 'w:utf-8') do |f|
	f.write(html)
end

By the way, this code works also with JRuby, and the only source I had to adjust was the following line:

1

RUBY_PLATFORM == 'java' ? string : string.force_encoding('utf-8')

JRuby is able to handle the utf-8 format way better.

Soon it turned out that it was easy to setup a quick prototype which contains only a subset of functionality of the final version. I’ve worked on iBox3D in my spare time as often as possible, but also I had to take some creative breaks. The more functionality I’ve built in the game the harder the work was getting. I’ve learned that a finished and polished product needs a lot of time, more that anyone would estimate. But after I’ve set a clear target to myself I could concentrate to get there. And finally I’ve managed to create a game for iPhone / iPod Touch and iPad.

iBox3D can be downloaded for € 0,79 / $ 0,99 in the Apple Appstore.

Also the game has now its own website which can be found here:

http://www.ibox3d.com

I appreciate your feedback.

Two problems are solved in this article:

• Install bowline on another OS then it was developed on
• Run bowline with another version of Titanium Desktop then it was developed on

Bowline was developed on Mac OS X and uses the version 0.4.4 of Titanium. I have installed it on Vista and tried to run the bowline-twitter example with Titanium 0.6.0. This is how I installed it and how I managed to solve the problems:

1. Install bowline
To install bowline ruby gems can be used:

gem install maccman-bowline --source http://gems.github.comI’ve got some dependency errors. E.g. I had now active support and newgems installed. Install the other gems first (I think gem has also a dependency switch –include_dependencies or similar – I’m not sure).

2. Install Titanium Desktop
Donwload it from http://www.appcelerator.com/products/titanium-desktop/ and execute the installer.

3. Get the bowline-twitter example from github
You can get the bowline-twitter example from this URL: http://github.com/maccman/bowline-twitter/tree/master. After unpacking I renamed it to “twitter” and saved it c:/dev/bowline/twitter.

4. Building the twitter client
Open a command line, change to the project directory (c:/dev/bowline/twitter in my case) and execute the following command:

rake appI’ve got this error:

C:\dev\bowline\twitter>rake app
(in C:/dev/bowline/twitter)
rake aborted!
Exec format error - C:/ProgramData/Titanium/sdk/win32/0.6.0/tibuild.py -d C:/dev/bowline/twitter/build -s C:/ProgramData/Titanium -a C:/ProgramData/Titanium/sdk/win32/0.6.0 C:/dev/bowline/twitter/build/appBackground: bowline was created on Mac OS X. On the Mac console, which is called bash, files have a line which tells the OS which program should execute them. E.g. the run file in the folder twitter/script:

#!/usr/bin/env rubyrequire
File.join(File.dirname(__FILE__), *%w[.. config boot])
require "bowline/commands/run"From the first line Mac Os knows to start this file with ruby. So in the command line you can just type:

script/runIn windows you have to type:

ruby script/runWindows doesn’t recognice first line. In the build script for bowline a python script should be executed but windows doesn’t know that it is a python script. Step 5. will explain how you can change it.

5. Changes in the build file app.rake
This file can be found in

%ruby_home%\gems\1.8\gems\maccman-bowline-0.4.6\lib\bowline\tasksLine 146 and following reveal that a python script tibuild.py is called. This is part of Titanium Desktop. So you have to install Python. I’ve got Python 2.6 installed. Please also make sure that the Python path is also in you environment variable PATH (Control Panel > System > Advanced System Settings > Environment Varibales > PATH). E.g. my path variable contains besides many other entries C:\dev\Python26\ . Remember that different paths have to be seperated by ; without space.

Back to app.rake. Find the following line (146):

command = []And now add another line:

command = []
command < < "python"Now the command line error is solved. But there is another thing which has to be changed in app.rake. The python script mentioned above also parses the manifest file of the project which contains meta information, like the author, name and dependencies. It also determines the required runtime version of the project. The twitter client was built on Titanium Desktop 0.4.4, I have 0.6.0 installed. This leads to an error, so I had to change this in the app.rake.

I replaced all versions from 0.4.4 to 0.6.0 in app.rake.

6. Try to build it again
Command Line:

rake appStill an error:

C:\dev\bowline\twitter>rake app
(in C:/dev/bowline/twitter)
Couldn't determine your source distribution for packaging runtime version 0.4.4. Please specify.Still the version error. If you open the manifest in twitter/build/app/ you can see that there is still the version 0.4.4 inserted. So you have to regenerate the manifest (step 7).

7. Regenerate the manifest
Command line:

rake app:configure8. Final build
Command line:

rake appThis builds you app and is not necessarily needed if you just want to run it.

9. Run the twitter client
Command line:

ruby script/runWorks!

Bowline is very interesting but also very young. I hope that the Author, Alex MacCaw, will drive the development further so that it really gets usable.

Links:
Blog Post of Alex MacCaw
Blog Post on RubyInside
Bowline on github
Bowline-Twitter on github
Titanium Desktop