Thursday, October 20, 2016

JavaScript: Callbacks, EventEmitters, and Promises - which one to use?!?

Short version

If you have something that's simple and always synchronous, don't use any of them. Just write a dumb function.

If you have a function that's simple and only needs one asynchronous response - and there are no other potential responses - then a callback is fine.

If you have some kind of object that could have several different potential asynchronous responses - at various different points in lifecycle - and you might or might not want to listen to none, one, or more of them? Then use EventEmitters.

And finally, use Promises when:

  • You have a collection of asynchronous functions, and you need to respond only when all of them have returned, or any one of them have returned.
  • You're doing mostly 'imperative' functions and don't need to pass a lot of values around, you just need to chain together some callbacks in sequence.
  • You have some functions that might be synchronous, and some that might not be, and you're not sure which until runtime
  • You have a collection of asynchronous events that are all firing, but the order that they must complete in is dependent upon some value determined only at runtime.
  • (weaker argument) You are falling victim to callback-hell, and your code is steadily creeping rightward

Long version


function foo(param1,param2,param3) {
  return "something";

var a=foo(1,2,3);

If you can do it this way your life will be better. Do it this way if at all possible.

Simple Callbacks

function foo(param1,param2,param3,callback) {
  process.nextTick(function () {

foo(1,2,3,function (result) {
  console.warn("Yay, we got result! "+result);

If you find you're passing "callback1, callback2, callback3" definitely don't do this. But for small, simple asynchronous functions, with not much else going on, this is still fine. Still pretty easy to reason about. As functions grow larger, and nested callbacks grow deeper, it gets harder and harder to reason about, though. The invocations of your little function probably ought not to be more than just a few lines; if they are, you should consider the next option...


I think 95% of the EventEmitters I create end up being 'classes' that extend the EventEmitter class, and I think that's probably a good way to do it.

   /* ..... */
   /* ..... */
   /* ...... */ 
   /* ...... */
   this.emit("complete","something",success === true);

What's great about this model is that someone who's consuming this object might only care about one particular event - in which case they can listen for just that one. I believe it's ok, and actually good, to emit liberally, even events that are similar but not the same (in my example "success"/"failure" as well as "complete").

Another nice side-effect is that all of your various listen events (.on(foo)) help document what the callback is actually for. E.g. -

on("complete",function (param) {
  /* see? Now we know this event handler fires when things are complete! */

If you're not careful, you can absolutely slide into callback-hell here. But this is my personal favorite pattern to use. It's pretty extensible.

Never do synchronous callbacks; ever. If you want to do something 'immediate' at least wrap it in a process.nextTick(function () {/* blah */}); block; that'll effectively be immediate but allow for someone to use it in the way most EventEmitters are used.

Never throw errors; just emit "error" instead.


These are massively over-hyped as the solution to everything. While they are actually very, very cool; they definitely have some real drawbacks.

  • They can get hard to debug
  • They can be confusing
  • missing something like a return - which is super-easy to do - may just cause silent code malfunctioning instead of issuing any kind of error
  • Propagating data forward from previously-resolved promises into later promises looks and acts weird.
  • You lose a lot of the benefits of 'closing-over' variables
But, when used properly - they can turn something nasty like this:

foo.on("bar",function (baz) {
  bif.on("blorgh",function (bling) {
    bloop.on("gloob",function (fweep) {
      /* .... */

Into something much prettier like this: (baz) {
  return "thing";
}).then(function(bling) {
  return "other_thing";
}).then(function (fweep) {
  return "last_thing";

Which, especially if you end up with a super-long list, can be helpful.

You can also use .catch() to grab any error in your list of actions - and that can be enormously useful.

Also, if you have an array of promises, you can do something like -

Promise.all(my_array_of_promises).then(function (results) {
  /* do something */

Which can be very, very handy.

Where it starts to get ugly is, if in that example I gave above - if you need to treat an error condition for each of those steps slightly differently. Now you can throw various .catch statements after each .then statement, but I can imagine trying to visually read through that as being a nightmare.

The other huge thing here is that some promises can be fully synchronous - e.g. Promise.resolve(7) - that's a promise that will resolve to the number 7. And some promises (well, probably most of them) are asynchronous. This is great, and the ability to unify these two modes together can be very helpful.

So, absolutely use them when they make sense. But my current thinking (which might change) is that you should use the simplest asynchronous mechanism that expresses what you need, without adding complexity. Step up the complexities of your tech as you need to, but not before.

Use the right tool for the job.

Monday, August 31, 2015

A pinko-commie lefty-liberal's _technical_ argument against The Secretary of State using Her Own Email Server

If an employee of mine tried to send email from their own server, instead of using my company's, I'd reprimand them, harshly. And if they continued to refuse to use the appropriate email server I'd have to terminate them.

Why? Security.

Who ran Hillary Clinton's Email server? What software did it run? Where was it hosted? How physically secure is that location? How secure is the ISP that serves data to that server? The administrators of that server - are they background-checked? Are we sure they are not in the service of any foreign entity, or even a domestic one? Is the server patched to the latest version? Are there any vulnerabilities on the version it is running? Are there any backdoors or rootkits or other such stuff on there? What's the update schedule on that server? Is there any strong cryptography on it? What protocols are running?

And the answer in just about all of these cases is, "I have absolutely no idea." And that is completely and totally unacceptable.

If you administer an email server, YOU CAN (usually) READ ANY EMAIL THAT IS ON IT. When it's some regular schmoe somewhere, that's less of a big deal (but still something to be concerned about). When it's the Secretary of State of the United States of America, that's a bit of a bigger deal.

If you are reading this and feeling like I'm not exactly right, go talk to your email admin. Ask them to read you your latest email - "just so you can see if your email software is working." Let me know what you find.

Saturday, May 30, 2015

How to do: Gulp + Browserify + Coffeescript + Sourcemaps + Uglify as of mid-2015

I blew so much time on this, it's crazy.

I got it down to something simple and clean, finally. And it turns out the "recipes" section on the Gulp website pretty much had the answer already. I just needed to understand more about what I was trying to do in the first place.

Note: You need to install coffeeify to have it available as a transform!

var gulp        = require('gulp');
var browserify  = require('browserify');
var source      = require('vinyl-source-stream'); //to 'rename' your resulting file
var uglify      = require('gulp-uglify');
var buffer      = require('vinyl-buffer'); // to transform the browserify results into a 'stream'

var sourcemaps  = require('gulp-sourcemaps');

gulp.task("your_target",function() {
      entries: ["./"],
      debug: true,
      extensions: [".coffee"],
      transform: ["coffeeify"] // npm install --save-dev coffeeify
    .pipe(sourcemaps.init({loadMaps: true,debug: true}))
    .pipe(uglify(/* {
          debug: true,
          options: {
            sourceMap: true,
      }*/)).pipe(sourcemaps.write("./" /* optional second param here */))


If you don't want the sourcemap URL in your resulting JS file (which I didn't, because I keep my sourcemap file private), add a second parameter ,{addComment: false} to the sourcemaps.write line near the end.

Edit: - the parameter to uglify isn't even needed.

Saturday, January 10, 2015

Debugging or Troubleshooting the PHP Autoloader

I used PHP pretty heavily a while ago. It's still - in its primitive, non-objectey, non-frameworked form - one of the environments in which I can 'think' in the language the fastest. If someone had a gun to my head and said "build me this thing 'x'" - I would probably do it in PHP (unless it were large enough with lots of forms/fields/tables/etc., then I might use something else. Or maybe still PHP? I dunno.)

But that means I missed out on the newest revolutions in PHP - composer, Laravel, all that stuff. And some of it is actually pretty cool, once you get a handle on it.

Composer is very similar to how Bundler works under Ruby, or how npm works under Node. You say which libraries you need and which versions in a config file, then say 'composer install' (or 'php composer.phar install') and it magically goes and gets them and installs them. Pretty neat.

One of the more subtle pieces you get is no longer having to say require "foo.php"; - the new system of Autoloading does that for you, if you've gotten everything set up right. It seems to be powered by PHP Namespaces - each composer-able package declares which namespace it's going to set stuff up in, and then if you try and use something from that namespace, it automagically gets loaded up. Pretty neat!

It's actually very neat. doesn't work. And then I started to find that it was extremely difficult to troubleshoot what it's actually doing, and why my custom package wouldn't autoload.

Probably the first place to start should be the DebugClassLoader component ("/Symfony/Component/Debug/DebugClassLoader.php"). That was installed by default in the Laravel app I was playing with. It looks like there's a nice static class method you can invoke, which will help troubleshoot what's going on in the class-loading process - "DebugClassLoader::enable()". There seemed to be another Debug component - somewhere - that wanted to enable that feature - "/Symfony/Component/Debug/Debug.php" - but I couldn't end up figuring out how to turn the damned thing on. If you do? I bet that'll be more helpful than the other stuff I ended up playing around with. That DebugClassLoader seems like it will wrap the autoloading process and yell at you if it can't find the classes it wants in the appropriate files, and there's not a lot else going on with the autoload sequence than that.

But, due to my ignorance, the most useful tool that worked for me was manually loading up the autoloader object and just asking it dumb questions:

$autoloader = require "./vendor/autoload.php";

$results = $autoloader->findFile("\\MyNamespace\\MyClassName");

print "Found file for class at: $results";

Other methods I ended up playing with were:



Enough poking around with some of these techniques eventually got me to the point where I could figure out what it was that I was doing wrong. Hopefully I can save someone else a few hours of heartache too.

Sunday, August 17, 2014


So there are a few people I've recently met who are anarchists, and I've told them all that I disagree with them. But I wanted to lay down my explanation as to why.

Let's not talk about the moral underpinnings - because the morals behind any socio-political-economic system are always super-duper good and just. (e.g. socialism's "From each according to ability, to each according to need"). But the devil's always in the details. So let's get into some details.

There are examples in actual history we can look at. The best modern example is probably Somalia, which basically has had no functioning central government for decades now. It, by most accounts, is not a very nice place. It is ruled by warlords. It is crippled by poverty and food shortages. If anarchy were so great, why isn't Somalia a great place for anarchists to live?

We know the way that power tends to aggregate. See organized crime, or large multinational corporations (or perhaps I repeat myself - ZING). Though a great Libertarian/Anarchist argument against the organized crime part is that organized crime got the biggest boost in power during Prohibition. And the Mexican drug cartels that are currently dominating Mexico are being substantially weakened by the legalization of marijuana here in the US. And it's a very good point, but the truth is that organized crime existed before and after prohibition, and will still exist even after we legalize pot. And large corporations existed before anti-trust legislation came about in the late 1800's, early 1900's, and afterwards. (Again, another Anarchist argument might be that large corporations would not have as much power without some kind of government intervention - if so I'd love to hear more about that; I think it was true of the Dutch East India Company, but more examples would be even better)

Let's talk about a small town operating under anarchism. We'll completely ignore the problems inherent in a large city - like my own New York - and just start focusing on my example small town. It's got 100 residents, let's say. Large cities will be probably even more problematic but I think I can explain my issues with my small town.

Problem #0 - I can walk down the street and just shoot someone in the face. There is no 'legal' ramification to that. If the person I do that to is not well-regarded, people might even cheer me on! Of course, if I do that to some beloved town local, I would assume that someone might come back and shoot me in the face. And I don't want that. Of course the trick is to kill someone when no one else is looking.

Problem #1 - just about everyone has to own a gun. Some people might not, but in general, you just need to own one, primarily as a deterrent. With no formal social safety net, (plenty of informal ones, mind you! But nothing that's guaranteed to catch people who are down on their luck) - there will be very desperate, very poor people who need things; or very depraved and lawless people who will take what they want. Some people may have an issue with having to own a firearm - and a system that practically forces them to do so seems unfair to those people. So a system that does not approve of force is now inherently, due to its structure, forcing people's behavior.

So eventually due to the Organized Crime/Large Corporation problem, you will have to step up from the everyone-is-armed-at-home problem, and in to the Defense problem. E.g. instead of one down-on-their-luck person trying to take your possessions, killing you in the process - you now have the potential for a gang to come roaming through your town and ransack the place. You need some kind of defense; an army. So you hire one - and this is Problem #2. Well, it's problem 2, 3, 4, 5 and 6. First off, you have to find an army that's willing to defend your town - and we have a perfectly free market, so there will be a lot of competition, right? Maybe. In fact, your roving-gangs-of-ransackers are just as likely to be the 'army' that you'd hire. Or be somehow in cahoots. So how do we pay these people? We have 100 people in town, and we need to have them all band together to pay the army. But Old Man Caruthers doesn't want to pay! Well, we can't force him - Non-aggression principle. Now we have problem #3. So then we have to increase the price that everyone else pays to cover his share - and now all sorts of other people are going to start balking at the prices. So eventually you have to say, "either you pay, or you can get out of town." That sounds like force. Or maybe you make a deal with the army - mark the houses that have paid, and they get protection, and the ones that don't, don't. Sounds like a mess. And things like securing the town's borders won't work in that way.

And how did we manage to select which army we got to defend us? A vote? A vote where only consensus is allowed? At 100 people consensus will be hard. At 1000 it will start to become impossible. As soon as we start having a 'majority' - then we're coercing people, and breaking our own rules. Problem #4. (What about payola; the guys in Army Group #1 slipping $100 each to the people who are 'on the fence' to secure their vote?)

Problem #5 - who is to keep our 'army' in check? Let's say I've got a roving band of raiders. Why don't I meet up with the person in charge of the army-for-hire, we sit down and have a nice lunch, and I offer them a huge cash payout to stand down on such-and-such a day? Well, certainly, that would erode the trust one might have in such an army - if word ever got out. But why would it? My raiders would just go and kill everyone.

Problem #6 - how do you fire your army? Ideally, with another army, and the first ones just leave. But what if you decide you just don't want an army at all, then what? And what if "your" army decides they don't want to be fired?

And we haven't even gotten into policing yet - which would probably end up being problem numbers 7 through 15...

And we haven't even figured out what currency any of this stuff is bought or sold in. More problems.

So I think the real, fundamental economic problem here is this:

A market with no regulation at all is not at all free.

Not everyone has perfect information to make perfect economic choices. Certain goods and services exist in certain locations, and cannot be quickly or cheaply transported to wherever they are needed. Monopolies, cartels, and collusion happen and drive prices up. Gluts happen and drive prices down. There is inherent friction in every economic transaction.

And the political problem is this:

The Tragedy of the commons.

Without the ability to coerce people, and without the ability to form majority rule instead of consensus, you aren't going to be able to do anything as a society. "The Commons" doesn't have to be a physical thing; like a stream or a pond or grazing grass - it can be like our 'how do we pay for the army' problem above. Private property is not a solution. Private ownership of a common good like the water supply runs you into problems with inelastic demand - everyone needs water, so why not jack up the price for access to it? Still more thorny problems.

The Social problem is this:

This system completely and totally shafts the poor, and rewards the rich.

Can't afford to pay for the army? Get out of town - or get treated however Mr. Caruthers got treated, above. Down on your luck? Hope for some handouts from private individuals. Still starving? Die. How much does this society help lift up the poor? How much does this society prevent the rich from just becoming more and more massively super-rich generation by generation; just sitting idle, reaping the rewards of actions done generations ago, or reaping the rewards of simple dumb luck?


Some interesting pro-anarchy thoughts I had while writing this up: What if you were to view this government as exactly the final result of having one of the armies in problem #2 defending you? E.g. the army won't let you choose another army, it forces you to pay it. Though, to be nice, they charge a lower percentage of income to poor people and a higher one to rich people. What if that is, in effect, the government we have now, and modern taxation?

The other interesting one is how mafias and black markets tend to disappear when everything is permitted. Organized crime was at its most powerful here in the US in the middle of prohibition. Right now, it deals in drugs and other 'sinful' things. If all of those things were permitted, would organizations such as these disappear? What purpose would they serve? <CAVEAT - ARGUMENTUM AD MOVIE-UM> - in the Godfather Part II, we see a little glimpse of the early Sicilian Mob - and, while they were certainly murders, thieves, and extortionists - they were also community-builders, who helped their communities when the government would not. Maybe organizations/groups/towns whatever might end up acting like that?

I was also going to use the metaphor of prison for what happens when you 'have no rules'. But prison has tons of rules! Yes, but the guards are really keeping "animals in cages" - and may not necessarily care for what the "animals" do. So that might-makes-right, everyone grouping into 'tribles' environment might be what you end up with. But what if that's what the US *is* - the 'rules' the government puts on us are the prison guard's rules, and today's society is the same as that prison - tribalism, might-makes-right, what-have-you? I think the metaphor breaks down, but I still think it's interesting.

Friday, June 13, 2014

Some actual things you could do for gun control stuff

Most people would probably agree to some formation of the following:

Someone ought to be able to own a firearm to protect their home and family. And all of these horrible shootings that keep happening are awful, and we should try to prevent them from happening. We can't stop them all, but we can at least try to make it more difficult for them to happen.

And if you do agree with that, here's my proposal:

Background-checks. Yes, even for private sales. In an era of $100 smart-phones, there's a way to do it. When you sell a car, someone has to fill in or file a registration. It's not unreasonable.

One-way database. In the same way you can have a 'hash function' which can map from source data to a hash value (But *not* backwards!), you should be able to map from serial numbers to people. But not the other way around. If you really need to see if someone has a firearm, you can get a warrant to search their house. But if a gun is used in a crime, and the serial number can be read off it, we need to be able to figure out who that gun belongs to. I would want to appoint some kind of privacy advocate to protect this data as well. The idea of the cops running around after Hurricane Katrina happened, confiscating legal firearms of civilians is something that should be prevented from even happening, and made even more terribly illegal than it was already.

Providing a firearm to someone who then commits a crime means you are an accessory. And should be criminally charged. Improper storage or securing of ones firearm(s) which are then used in crime mean you are negligent, or possibly an accessory.

And yes, that means if you 'lose' or have your firearm stolen, you need to report it. And that means if you didn't secure it you can be charged. And that means you need to check in on your firearm every, say, 6 months or so - no saying "Oh, I forgot I had it! Haven't looked at where I keep it in a while..."

That also means when you're doing tearful press conferences about how no one knew that your kid could go shoot up a school, it's likely you'll be wearing prison-orange. Because you probably weren't properly storing your firearms. Because if you were, maybe your kid would've had a harder time shooting up that school?

As for the definitions of what these things are? (How would a 'household firearm' work? What does 'properly stored' entail? Etc.) I don't know and I think that's probably an important place for us to get to. But let's start a conversation here.

What about firearm types? I don't care about that. AR-15 or AK-47 or simple 9mm Glock. Honestly, that's the wrong road to go down. The right road to go down is stopping people who shouldn't be able to buy guns managing from buying or acquiring them somehow. And making gun-owners responsible for secure and safe storage of their firearms. That, honestly, is not unreasonable.

Friday, April 11, 2014

Running Riak in Classic-EC2 (in a non-moronic way)

So I've been playing around with Riak lately - and it is a spectacular piece of software. It has a lot of the cool clustering goodness that you might get from something like Cassandra - but is a lot less of a pain in the ass to administer and deal with. Interacting with the database in an ad-hoc fashion is actually delightful - you can use lovely little REST URL's like: to fetch one particular key. Riak doesn't care what you store there - you could certainly throw a JSON blob in there, but you can throw whatever else you might like too. Want a secondary index? Make sure to use a 2i-capable backend (Leveldb, or others), then declare what values you want indexed when you write your data. Riak doesn't load-balance over your cluster instances, but there are a ton of perfectly good load balancer solutions that already exist that work great with it. And if you need something tighter/faster than HTTP REST URL's, you can use their protocol buffers interface for something that's more socket-ey. If you're into that kind of stuff, check it out. I'm really excited about it.

But be wary if you're running it in Classic AWS (though their advice on running in VPC seems solid). The advice they give on their documentation website is terrible and wrong-headed. They actually expect you to manually go in and reconfigure the nodes on your cluster if one of your instances has to reboot. I worked out a better way. Of course, I offered it back to them, but they have yet to take me up on it.

You should basically run it the same way you'd run any other long-lived server in AWS. Allocate an Elastic IP address for each node. Look up the cool "public DNS name" for each of your elastic IP's (should look like: ""), and use either that or a nice CNAME that points to that as your host identifier. That way, when your instances inside AWS resolve the name, they get the inside IP addresses. If your instance has to reboot, just reassociate your EIP to the instance. And that's it. Oh, and the bits in the config where you're supposed to put your internal IP address as something to listen to? Just put, it works fine (though there are allegedly some possible problems with Riak Control and HTTPS that way; that's what Basho tells me but I don't really know anything about that).

And you should of course protect your Riak port the same way you'd protect any other port for any other internal-facing system. There. That's it. I have shut down and restarted my entire cluster (forcing it to get brand-new IP addresses), and using this method it seemed to work just fine.

The method Basho proposes to handle a reboot is as follows: Stop the node. Mark it 'down' using another node. Rename the node to whatever your current private IP address is. Delete the ring directory. Start the node. Rejoin the cluster. Invoke the magic command "riak-admin cluster force-replace " (Oh, I hope you remembered to keep track of what the old name was when you renamed it!) (Oh, and one more thing! Up until a few months ago that was misdocumented as "riak-admin cluster replace ..." which would toss your node's data). Plan, then commit the changes. If you like to do it that way, you are some kind of weird masochist. And if you think this is remotely a good idea, then you are not good at technology.

I got into a long back-and-forth with one of their engineers about their crazy recommendations, but he refused to budge. I even submitted them a pull request with a fixed version of the documentation - and they still haven't merged it. Why? I have no idea. The general impression I got is that the guy I was talking to was just being obstinate. We were literally talking in circles. "Why don't you just use VPC!?" I'm sorry dude, I can't, because all my shit is on Classic and that's where I'm stuck, for now. "But if you give it an elastic IP, now it's accessible to the world!" No more or less so than if it just picks its own IP, buddy. "But you'll be trying to talk to the outside IP!" No, those funny names resolve to inside IP's when you're inside of Amazon. "Well, you should just use VPC!" And so on. Literally, this dude told me to use VPC like 5 times in the course of our email exchange. When I was explaining how to use Riak in Classic-EC2.

So, yeah. Good database. Had a really nice leisurely chat with one of their sales guys, too - he seemed really cool and knowledgable and low-pressure. But this 'evangelist' fellow certainly makes that group seem pretty dysfunctional to me.