Skip to content

Printing only the matching column with awk and grep

I’ve spent way too many minutes looking this up over the years.

awk 'match($0,/regexp/) {print substr($0,RSTART,RLENGTH)}' inputfile

and about 1 time in 10 I’m startled to remember grep makes this very easy…

grep -o regexp inputfile

From here.

Posted in Uncategorized.

No comments

By daemianmack – October 7, 2011

HATEOAS with visuals

This is the most satisfyingly visual explanation of HATEOAS I’ve seen.

Here’s what I came to understand. Every communication protocol has a state machine. For some protocols they are very simple, for others they are more complex. When you implement a protocol via RPC, you build methods that modify the state of the communication. That state is maintained as a black box at the endpoint. Because the protocol state is hidden, it is easy to get things wrong. For instance, you might call Process before calling Init. People have been looking for ways to avoid these problems by annotating interface type information for a long time, but I’m not aware of any mainstream solutions. The fact that the state of the protocol is encapsulated behind method invocations that modify that state in non-obvious ways also makes versioning interesting.

The essence of REST is to make the states of the protocol explicit and addressible by URIs. The current state of the protocol state machine is represented by the URI you just operated on and the state representation you retrieved. You change state by operating on the URI of the state you’re moving to, making that your new state. A state’s representation includes the links (arcs in the graph) to the other states that you can move to from the current state. This is exactly how browser based apps work, and there is no reason that your app’s protocol can’t work that way too.

Also, Martin Fowler’s article Richardson Maturity Model: steps toward the glory of REST is excellent. Richardson’s book, Restful Web Services, doesn’t mention his own maturity model or HATEOAS itself — it gives the most passing of references to Link tags or headers, for example. Good book, though.

Posted in Uncategorized.

No comments

By daemianmack – July 15, 2011

Camphone thread engine

A couple years ago, I built a small Ruby script to let users send pictures from their phones to a webforum. This was an idea I’d been kicking around for a while since the early mobile photo-sharing sites sprang up, and when I stumbled across the MMS2R gem, I figured it was time. It quickly became one of the more popular features of the forum; it’s neat to be able to chime in from where-ever you are with a quick snapshot of what you’re seeing or doing.

The other night, I needed to dig back into that script and found my brain itching from the way the script had organically grown a bit sideways over time, so I decided to refactor it and toss it up on Github in case it’s helpful as a starting point for anyone who wants to do something similar.

Posted in Development.

No comments

By daemianmack – June 22, 2010

magit Cheatsheet

magit is an emacs mode for git, and is crazy awesome. I’ve distilled the Magit User Manual into a command-centric cheatsheet for my own purposes and offer it here in case you find it helpful…

Magit Cheatsheet

Also including the upstream org-mode file in case you feel like wrestling with org-mode’s org-export-as-html and its weird set of quasi-controls.

In order to get org-mode to export that nicely, I had to add two user variables to my .emacs, along with a macro I used to quick-replace the TD elements with the classes they need. Here’s the snippet.

Posted in Development.

No comments

By daemianmack – September 24, 2009

Some performance monitoring stuff that’s been on my radar lately

Monitoring server load on Linux

I’m all about top and mytop, but there’s at least one surprise in here for me — namely, /usr/bin/tload.

Measuring & Optimizing I/O Performance using iostat…

If IO performance is suspect, iostat is your best friend. Having said that, the man pages are cryptic so don’t be surprised if you find yourself reading the source. To get started, identify the device in question and start a monitoring process…

which contains a note on how Facebook stores 360 photo uploads every second.

Posted in Server.

No comments

By daemianmack – June 24, 2009

astalavista.com compromised

Anybody who’s had an interest in security over the last 10 years has probably heard of astalavista.net — kind of a clearinghouse for all sorts of security-related info, vulnerability announcements, etc.

Here’s a blow-by-blow transcript of their systems getting completely wiped out by someone who didn’t agree with their practices. The transcript is hosted on pastebin.com but was originally available as the index.html of the site itself.

It looks like they used custom code to exploit a hole in astalavista’s Litespeed webserver; googling the exploit used just brings up copies of the transcript itself.

This is probably the most brutal exploit transcript I’ve ever seen — reference line 1839 where the exploiter, having found where the backups are being stored by trawling the admin’s .bash_history, FTPs there and deletes their offsite backups. (If your production system stores the FTP password or password-less SSH keys to your offsite backup server, exposed through your .bash_history no less, is it really offsite?)

astalavista is still down at the moment; I imagine they’ve scrambled to change all their passwords, shore up their exposure, and see if they can dig up a backup from anywhere.

It makes a pretty remarkable contrast with this recent How I Hacked Hacker News (with arc security advisory) accounting by someone who compromised Paul Graham’s news.ycombinator.com via some pretty painstaking research of system behavior — and then worked with Robert T. Morris (the Robert T. Morris) to get the issue fixed.

Posted in Server.

Tagged with .

No comments

By daemianmack – June 5, 2009

HOWTO: Rack Middleware for API Throttling

I will show you a technique to impose a rate limit (aka API Throttling) on a Ruby Web Service. I will be using Rack middleware so you can use this no matter what Ruby Web Framework you are using, as long as it is Rack-compliant.

Awesome. I can think of so many uses for this basic pattern to replace stuff I usually have to do at the app level (slow) or at the Apache level (difficult to port, prone to decay).

Comments note that Memcache might be a better pick than Redis, which works even better with the setup I have in mind, though Redis sounded fun too.

Posted in Server.

Tagged with , .

No comments

By daemianmack – May 15, 2009

HOWTO: Totally Unmathematical Chili

Here’s the basic skeleton of the recipe I use for chili. I started a few years ago with something called “Touchdown Chili” that a co-worker printed out of an online magazine, and over the years I’ve tweaked it relentlessly. I still have the original print-out that I pull out and use as a rough guide, even though my version has changed so much the printout is basically a series of phase cues — i.e., “time for spices”, “ok, do the meat”, “now you should be about finished adding things”, etc.

Part of the joy of this aboriginal approach is that it’s different every time.

A note on the “Do beans belong in chili?” holy war: Personally, I have no particular use for pulses, though sometimes I’ll make a sidecar batch for my fiancee (who is vegetarian) that includes a robust mix of black beans, white beans and kidney beans. She likes it. I make no specific recommendation in this regard and consider this momentous personal decision best resolved via quiet introspection and discreet consultation of sympathetic clergy members or other local community leaders.

Continued…

Posted in Random.

Tagged with .

No comments

By daemianmack – April 28, 2009

Making emacs’ word navigation respect underscores everywhere

I’ve been playing with emacs lately, using technomancy’s emacs starter kit. It’s awesome so far, but some piece of the (rather large) config fileset is clobbering what I’d consider normal navigation behavior.

I work with a ton of stuff that uses underscores, especially SQL, and it’s been driving me crazy that, on its own, my emacs config doesn’t consider the underscore punctuation in all modes, so hitting M-DEL (Alt+Backspace) to delete back one word in sqlplus mode deletes the whole string — underscores and all.

That is, if | is our cursor, and we’re inside emacs at a sqlplus prompt like so…

desc prod.some_long_table_name|

hitting Alt+Backspace once gets you…

desc prod.|

which is annoying, especially because emacs’ autocomplete is so awesomely useful in sqlplus mode. I’d guess most database schemas have lots of tablenames that share prefixes (e.g., user_history, user_bookmarks, user_buddies, etc), which renders that autocomplete somewhat less than effective.

I’d much rather have it so that the cursor stopped at underscores. On successive keypresses, you’d zap back another segment, stopping at the underscore, like so…

desc prod.some_long_table_name|
desc prod.some_long_table_|
desc prod.some_long_|
desc prod.some_|
desc prod.|

This is already the long-established behavior of interfaces that inherit from emacs, like BASH’s emacs mode, and irssi, but is evidently a legacy design decision within Emacs without an obvious config handle, nearly undiscussed as far as I could google…

From a comment in python-mode.el:

;; For historical reasons, underscore is word class instead of
;; symbol class. GNU conventions say it should be symbol class, but
;; there’s a natural conflict between what major mode authors want
;; and what users expect from `forward-word’ and `backward-word’.
;; Guido and I have hashed this out and have decided to keep
;; underscore in word class. If you’re tempted to change it, try
;; binding M-f and M-b to py-forward-into-nomenclature and
;; py-backward-into-nomenclature instead. This doesn’t help in all
;; situations where you’d want the different behavior
;; (e.g. backward-kill-word).

Luckily, TFM is rather nice.

The short answer is: one needs to specify the underscore character as a member of emacs’ punctuation class…

(modify-syntax-entry ?_ "_")

That will only work for text-mode, and gets overwritten on a major-mode switch by any new syntax table specified, so the following hook reference is necessary for this change to be global for all major modes…

(defun change-major-mode-hook ()
(modify-syntax-entry ?_ "_"))

Folks who aren’t emacs newbs might be aware of further subtleties, but this seems to produce exactly what I want — autocomplete and word navigation in every mode that’s identical to the other command-line interfaces I use.

Posted in Development.

Tagged with .

No comments

By daemianmack – April 28, 2009

TODO

HOWTO: Autocomplete fieldtypes in Expression Engine.
I recently had need to provide an autocomplete fieldtype in EE’s weblog UI. Nifty, and jQuery made it very simple. I need to package this up according to EE convention, document it, and release it to the EE community.

Rails to Expression Engine and back, from a data modelling perspective.
While there’s definitely something to be said for handing a customer a solution built on something as tried-and-true as mod_php, EE has made me realize how spoiled frameworks in general (and Rails in particular) makes us (me) when expressing data relationships. Hopefully EE 2.0 comes out and obsoletes this idea before I type it up, though I expect it to be not just fun but also educational for the whole family.

Posted in Development.

Tagged with , .

No comments

By daemianmack – April 19, 2009