Hackers are jelly-brained dicks

Like, what the hell is the purpose of trying to hack a personal blog site other than you have a really small pecker, and you need to somehow make yourself feel better about your pathetic life?

I remember a few years back, my father passed away, and we set up a small website containing his daily communion emails, and some dip-shit from Pakistan hacked it to just replace all the pages with a bunch of self agrandizing bullcrap.

Grow a fucking brain sometimes, assholes.

Rails 2.3 + Ruby 1.8 UTF-8 Hack

I work on a project which is currently still locked in Rails 2.3 running on Ruby 1.8 – of course, as years have gone by, more and more support for internationalization has come up, and now with emojis being part of the UTF-8 standard, and people people trying to use them in blog posts and comments and the like, I obviously encounter the fiasco that is trying to have Ruby on Rails on MySQL deal with this.

It’s been a mess.

In the end, I’ve just opted for a hack on the String class which gets used at the point that the model’s properties are assigned:

class String

  #
  # Converts multi-byte characters which use more than 2 bytes into HTML entities
  #
  def to_multibyte_html_entities
    each_char.map { |c| c.bytes.count > 2 ? "&#x#{c.multibyte_ord.to_s(16)};" : c }.join
  end
  
  #
  # Identical to #ord but properly supporting multibyte, like later versions
  # of Ruby
  #
  def multibyte_ord
    unpack('U')[0]
  end

end

Grails 3.3 with Servlet Filter

I had a scenario (I don’t really want to get into it) where I needed to implement a proper servlet filter which would run before the Grails dispatcher got involved in the request. The Grails documentation says it can be done, but the instructions are ridiculously vague, and seem more centered on plugins rather than just a webapp.

I finally managed to work it out, and I wanted to show the source code involved for posterity, and anyone else who might stumble upon this solution.

First off, create your filter in /src/main/java/myPackage:

package myPackage;

import java.io.IOException;
import javax.servlet.Filter;
import javax.servlet.FilterChain;
import javax.servlet.FilterConfig;
import javax.servlet.ServletException;
import javax.servlet.ServletRequest;
import javax.servlet.ServletResponse;
import javax.servlet.annotation.WebFilter;
import javax.servlet.http.HttpServletRequest;
import javax.servlet.http.HttpServletResponse;

/**
 * @author Nathan Crause <nathan@crause.name>
 */
@WebFilter(filterName = "testFilter", urlPatterns = {"/*"})
public class TestFilter implements Filter {

	private FilterConfig filterConfig;

	public TestFilter() {
	}

	/**
	 * @param request The servlet request we are processing
	 * @param response The servlet response we are creating
	 * @param chain The filter chain we are processing
	 *
	 * @exception IOException if an input/output error occurs
	 * @exception ServletException if a servlet error occurs
	 */
	@Override
	public void doFilter(ServletRequest request, ServletResponse response, FilterChain chain)
			throws IOException, ServletException {
		HttpServletRequest httpRequest = (HttpServletRequest) request;
		HttpServletResponse httpResponse = (HttpServletResponse) response;

		for (int i = 0; i < 100; ++i) {
			for (int c = 0; c < 80; ++c) {
				System.out.print("-");
			}
			System.out.println();
		}

		chain.doFilter(request, response);
	}

	/**
	 * Destroy method for this filter
	 */
	@Override
	public void destroy() {		
	}

	/**
	 * Init method for this filter
	 * @param filterConfig
	 */
	@Override
	public void init(FilterConfig filterConfig) {		
		this.filterConfig = filterConfig;
	}

}

Now the really poorly documented part – where the hell do you tell Grails to load the damned thing? Turns out you just use /grails-app/conf/spring/resources.groovy:

import org.springframework.core.Ordered
import org.springframework.boot.web.servlet.FilterRegistrationBean
import myPackage.TestFilter

beans = {
	testFilter(FilterRegistrationBean) {
		filter = bean(TestFilter)
		urlPatterns = ['/*']
		// we want all other Grails filters to have loaded first
		order = Ordered.LOWEST_PRECEDENCE
	}
}

Now, I originally opted for using pure Java for the speed, but it did expose the problem that none of my domain classes were accessible, which I did require. So, I re-implemented the filter as a Groovy class in the /grails-app/controllers/myPackage directory.

It’s also worth noting that you will be required to create a new hibernate session in order to perform queries.

package myPackage

import javax.servlet.Filter
import javax.servlet.FilterChain
import javax.servlet.FilterConfig
import javax.servlet.ServletException
import javax.servlet.ServletRequest
import javax.servlet.ServletResponse
import javax.servlet.http.HttpServletRequest
import javax.servlet.http.HttpServletResponse

/**
 * @author Nathan Crause <nathan@crause.name>
 */
class TestFilter implements Filter {
	
	FilterConfig filterConfig
	
	void doFilter(ServletRequest request, ServletResponse response, FilterChain chain) {
		for (int i = 0; i < 10; ++i) {
			println("-" * 80)
		}

		Account.withNewSession {
			println(Account.last().inspect())
		}

		chain.doFilter(request, response)
	}
	
	void destroy() {		
	}
	
	void init(FilterConfig filterConfig) {		
		this.filterConfig = filterConfig
	}
	
}

Project: PhotoHosting

Description

This is a very simple website offering uploading and sharing of 2 megapixel size photos, at a relatively high quality, social-media aware, and very hard to “steal”.

Motivation

A lot of free image hosting services are either overly ad-filled, or explicitly prevent uploading content which may be considered “NSFW”.

I wanted something less restrictive, less busy with ads (but still able to have ads on it), using some technologies which I would hope would facilitate fast I/O.

One principle was to try to always have the photo viewer coerced into visiting the website itself, as opposed to simply serving the image file.

This is accomplished by having the social media thumbnails relatively small, and a lower quality. Any attempt to actively open the image itself will always result in the webapp serving an entire page with the image embedded.

Technologies

I wrote this website is written in Java on the Apache Struts 2 framework accessing an HBase database.

Of the NoSQL databases, HBase seemed to be the only one which natively supports larger binary content, so it was a relatively easy option.

You can access the primary site here: https://photohosting.hostnucleus.ca/

To view an example of how the photos are presented: https://photohosting.hostnucleus.ca/show/cooked-chicken-on-white-plate-8de5b69c-020d-4480-addc-5d3af7013ba8.action

Update

Well, thanks to the complete cluster-fuck that is Java 11 (with it’s complete butchering of Java compatibility), and Apache’s lacklustre ability to move HBase away from Java 1.8, this project is officially dead. I simply cannot get the Tomcat webapp to talk to HBase.

So … yeah … fuck the OpenJDK “community”, and fuck Apache.

Project: Kalliope

Last year, I set myself the task of trying to actually produce a bunch of smaller web projects for s few reasons:

  • To learn some new frameworks
  • To illustrate my ability to learn there different frameworks
  • To showcase my skills in development in general

To that end, I thought I’d explain each project here in my blog, and the motivation behind each.

First up is Kalliope – https://kalliope.hostnucleus.ca

This is a “font server” with two primary end-goals – be able to arbitrarily upload a true type font, and have it produce web downloadable fonts, or to actively serve unlicensed fonts (in the same vein as Google Fonts).

The latter end-goal (as detailed above) was actually the primary goal on project inception – I wanted to have the ability to use a variety of fonts, but I didn’t want Google snooping the traffic on my sites (e.g. cookies, etc.). You can think of it as a replace for Google Fonts, really.

The other end-goal came about when the place where I worked had a client who had purchased some proprietary font, and wanted it used on their website. I didn’t want this font directly in the font server itself (as it requires a license), but all the leg-work had already been done to perform the conversion.

Interestingly enough, the original work on this was actually done for a completely different project – I had implemented a lot of the code within a project for generating print materials using SVGs. Since the SVGs needed to be displayable in all browsers, as well as the final PDF, I decided to use “FontForge” to do all the heavy lifting. It was only years later when developing a website where it occurred to me to break out the functionality into a separate web service.

It’s also a good project to learn and showcase the Grails framework.

Turning off Rails’ annoying “secret_key_base”

One of the most frustrating things with Rails is how they force shit down your throat when you don’t need or want it. Enter Rails “credentials” and their “securing” thereof. Where I work, we have no use for it. We don’t use Heroku or AWS. So why force me to set up a bunch of files or environment variables for something which I don’t want???

And setting “config.require_master_key = false” does not resolve the issue.

Well, here’s how I killed that filthy bitch:

In the file “/config/application.rb” add the following method:

    def secret_key_base
      'SOD OFF'
    end

There – now Rails can take their stupid “security” and shove it up their asses.

And now WP is annoying me. Awesome.

Alternative to Ruby’s const_defined? method

Today I wrote a bit of code in a base class which checks to see if a constant named “EXPORT_MAPPING” was defined. I’m not a Ruby expert, so off to Professor Google I go, and all the references I could find say to use the “const_defined?” method, thus resulting in the following implementation:

self.class.const_defined?(:EXPORT_MAPPING)

Now, since the intention here is to facilitate multiple levels of inheritance, I have a subclass called “ImageSlideshowComponent”, and a subclass of that “NivoSlideshowComponent”. ImageSlideshowComponent contains the EXPORT_MAPPING definition, with the intention of NivoSlideshowComponent simply inheriting it. Which is sort of does. We correctly see if it we do this:

NiveSlideshowComponent::EXPORT_MAPPING

However, invoking the following gives us “false”:

NiveSlideshowComponent.const_defined?(:EXPORT_MAPPING)

Clearly I’ve misunderstood the intention of the “const_defined?” method. It seems to only indicate if the class in question itself has defined the constant.

Here is the revised code block from the superclass which works correctly:

self.class.constants.include?('EXPORT_MAPPING')

Grails/Spring service autowiring

While working on a Grails project, I came across 2 interesting things:

  1. autowiring for domain classes doesn’t work, even if you follow all the instructions on how to enable it (either globally, or in the domain class definition itself)
  2. autowiring on abstract service classes doesn’t work (so if you’re implementing a data service with some custom functionality, your services won’t get autowired)

I’ve been able to fix both of these issues using a very simple utility class:

import grails.util.Holders

/**
 * Since Grails doesn't seem capable of autowiring into abstract classes, this
 * causes a bit of an issue when creating data services which do more than
 * just get/save.
 * 
 * All those context beans are available, so we just need to fudge the idea of
 * autowiring, and this class helps by providing a quick reference to the
 * singe long line required to get those Spring beans.
 * 
 * This also helps with domain classes, which have autowiring disabled, and
 * despite all the documentation saying the contrary, into which you cannot
 * autowire.
 * 
 * That said, this might prove to be a faster solution for use within domain
 * classes anyway since we don't have to worry about autowiring happening on
 * every object instantiation, but rather we just have a reference to the
 * singular instance with in the main context.
 */
class Beans {
	
	static def get(String name) {
		Holders.grailsApplication.mainContext.getBean(name)
	}
	
	static def $static_propertyMissing(String name) {
		get(name)
	}
	
}

Now, instead of setting up a service like this:

def myService

You instead define a getter method like this:

def getMyService() {
    Beans.myService
}

You can then use “myService” as you would’ve expected to under normal autowiring circumstances.

Gradle deployment script for Grails webapp

Please note the “Update” section below.

I haven’t had much success with finding useful deployment strategies and/or scripts for Grails anywhere. The extent of the documentation I’ve been able locate for deployment simply tells you to create a WAR and upload it to the servlet container.

Not terribly helpful if you want to run a formal process.

So, for my Grails webapps, I came up with this. I create a file in the “gradle” directory named “deploy.gradle” containing the following:

buildscript {
	repositories {
		jcenter()
	}
	
	dependencies {
		classpath 'org.hidetake:gradle-ssh-plugin:2.9.0'
	}
}

apply plugin: 'org.hidetake.ssh'

// the below is in it's own ext block because it can potentially be
// overridden by .deployrc.gradle in the user's home directory, and the scmUser
// might be required in the following ext block (where the repository is
// set up)
ext {
	scmUser = project.hasProperty('user') ? project.getProperty('user') : System.properties['user.name']
}

// please note that the file must end in ".gradle" or Gradle won't know how
// to process it, and you'll get a "String index out of bounds: 0" error
def rcFile = new File("${System.properties['user.home']}/.deployrc.gradle")
if (rcFile.exists()) {
	apply from: rcFile.absolutePath
}

ext {
	timeNow = new Date().format('yyyyMMddHHmmss')
	// TODO: the below should use a repository directly associated with the individual user
	scmRepository = "${project.scmUser}@my.site:/var/repos/${project.name}.git"
	runEnvironment = project.hasProperty('env') ? project.getProperty('env') : 'production'
	scmBranch = project.hasProperty('branch') ? project.getProperty('branch') : 'master'
	tmpDir = System.getProperty('java.io.tmpdir') + "${project.name}/${timeNow}"
}

apply from: "${project.projectDir}/gradle/deploy/${project.runEnvironment}.gradle"

task deploy {
	group 'Remote Deployment'
	description 'Deploys a branch from your SCM to a target environment on a remote server'
	
	doLast {
		println "Deploying from branch ${project.scmBranch} to environment ${project.runEnvironment}"
		
		ssh.run {
			session(remotes.role('webserver')) {
				// upload the WAR to the target server(s)
				println 'Uploading WAR file'
				put from: "${project.buildDir}/libs/mercury.war", into: "mercury/mercury-${timeNow}.war"
				// stop Tomcat so we can get this party started
				println 'Stopping Tomcat'
				execute 'sudo service tomcat8 stop'
				// if a pre-existing link exists to "current", remove it
				println 'Symlinking'
				execute 'if [ -f mercury/mercury-current.war ]; then rm mercury/mercury-current.war; fi'
				// link our new WAR to the name "current"
				execute "cd mercury; ln -s mercury-${timeNow}.war mercury-current.war"
				// Tomcat doesn't clean up extracted WAR file directories, so we must flush out the /var/lib/tomcat8/webapps directory
				execute 'if [ -d /var/lib/tomcat8/mercury ]; then sudo rm -rf /var/lib/tomcat8/mercury; fi'
				// Fire tomcat back up
				println 'Starting Tomcat'
				execute 'sudo service tomcat8 start'
			}
		}
	}
}

task deployBuild << {
	println "Building WAR for ${project.runEnvironment} environment"
	
	exec {
		workingDir project.tmpDir
		environment 'TERM', 'dumb'	// this prevents grails+gradle for displaying a second progress bar during the following invocation
	
		commandLine 'grails', "-Dgrails.env=${project.runEnvironment}", 'war', 'mercury.war'
	}
}

task deployCheckout << {
	println "Cloning ${project.scmRepository}#${project.scmBranch} for user ${project.scmUser} into ${project.tmpDir}"
	
	exec {
		commandLine 'git', 'clone', '-b', project.scmBranch, project.scmRepository, project.tmpDir
	}
}

task deployCleanup << {
	println "Removing temporary directory ${project.tmpDir}"

	exec {
		commandLine 'rm', '-rf', project.tmpDir
	}
}

deployBuild.dependsOn deployCheckout
deploy.dependsOn deployBuild
deploy.finalizedBy deployCleanup

Also in the “gradle” directory is a subdirectory named “deploy” where I have the files specific to the environments to which I can deploy, such as “staging.gradle”:

remotes {
	web {
		role 'webserver'
		host = 'my.site'
		user = 'webapps'
		identity = file("${System.properties['user.home']}/.ssh/webapps")
	}
}

Now, you will need to make some modifications to your main “build.gradle” script. First, add in the SSH plugin dependency, immediately after the “buildscript” section:

plugins {
    id 'org.hidetake.ssh' version '2.10.1'
}

And then in the area where you have all the “apply” instructions:

apply from: "${project.projectDir}/gradle/deploy.gradle"

Using the script above, I can deploy a particular branch from within my git repository to a specific environment thus:

gradle -Penv=staging -Pbranch=hotfix/1.2.3.001 deploy

It’s probably not perfect, but since I’m new to Gradle and Grails, I think it’s a pretty good start!

Update!

Alas the “org.hidetake.ssh” plugin has not kept up to date with the latest updates to SSH, and as such the above scripts no longer work, and I have abandoned any attempts to use Gradle to effect deployments. Instead, I have switched over to using Ansible to deal with all of this.

CM3 v5.10 on Ubuntu

This was a fairly difficult task to figure out (no documentation out there), so I thought I’d share how I ended up getting the latest version of the Critical Mass Modula-3 compiler installed.

Please note that I was working an an emulated 32-bit operating system, so if you’re working on a 64-bit installation, make changes where applicable.

  1. You must first download an actual binary copy of CM3. This isn’t actively mentioned anywhere in the source documentation, but you will notice that as you try to do any of the scripts, you will always fail before anything else happens. If you pay close attention, you will see the script is trying to run “cm3” … which is annoying, because it’s CM3 you’re trying to install :PGo to the URL https://modula3.elegosoft.com/cm3/ and scroll down to the section “Target Platform LINUXLIBC6”. I just downloaded the first file, so it’s the whole shebang.
  2. Extract the file and run “cminstall” – this will install the cm3 compiler in a place you wouldn’t necessary expect: /usr/local/cm3/bin – this means you won’t be able to invoke cm3 right away, so you must …
  3. Add the cm3 binary directory to the path thus:
    export PATH=$PATH:/usr/local/cm3/bin
  4. Next, clone the latest source respository:
    git clone https://github.com/modula3/cm3.git
  5. From the directory cm3/scripts, run the upgrade.sh script

And presto – your CM3 should now be upgraded all the way.

Update:

When trying to build all the additional packages (using “do-cm3-all.sh buildship“), even if you have ODBC drivers installed, the process will fail when building the “db” packages. To fix this:

cd /usr/lib/i386-linux-gnu
ln -s libodbc.so.2 libodbc.so

This holds true for the UI package, as well, specifically lixXaw and libXmu:

ln -s libXaw.so.7 libXaw.so
ln -s libXmu.so.6 libXmu.so