Node.js and Redis

Using Redis to scale Caches, Events, and Queues up and accross Node.js Applications

Ryan-Sandy Lee

Web     ryansandy.com
Github ryan-sandy

Overview

  • Definitions
  • Introduction
  • Caches
  • Event Emitter
  • Queue

Definitions

  • Scaling Up
    • Adding additional instances (copies) of the same application
  • Scaling Out
    • Adding functionality to different code bases

Introduction

  • Redis
    • No-Sql
    • In-memory data structure store
    • Strings, hashes, lists, sets, sorted set, bitmaps, hyperlogs, ...
    • redis.io
  • Node.js

Introduction

Simple Example


var redisDriver = require('redis');
var redis = redisDriver.createClient();

redis.set("forever", "forever-value", function (err) {
  redis.get("forever", function (err, val) {
    console.log(val);
    redis.end();
  });
});
          

            forever-value

Cache

  • Design Pattern
    • Store data with an expiration
    • Retrieve quickly
  • Commands
    • setex
    • get
    • expire

Cache


var redisDriver = require('redis');
var redis = redisDriver.createClient();

redis.setex("expires", 1, "seconds", function (err) {
  redis.get("expires", function (err, val) {
    console.log(val);
  });

  setTimeout(function () {
    redis.get("expires", function (err, val) {
      console.log(val);
      redis.end();
    });
  }, 1500);
});
          

seconds
          
					
null

Cache

  • Scale Up
  • Scale Out
  • Useful for session data
    • Express-session & connect-redis

Event Emitter

  • Design Pattern
    • Messages broadcast to zero or more listeners
    • Possible for message to be consumed more than once
    • Messages do not persist
  • Redis Design - Pub/Sub

Event Emitter Example


var redis = require("redis");
var subscriber = redis.createClient();
var publisher  = redis.createClient();

subscriber.subscribe("channel 1");
subscriber.subscribe("channel 2");
subscriber.on("message", function (channel, message) {
  console.log("message on %s: %s", channel, message);
});

publisher.publish("channel 1", "Hello Edinburgh.js");
publisher.publish("channel 2", "Hello Edinburgh.js");


message on channel 1: Hello Edinburgh.js
message on channel 2: Hello Edinburgh.js

Event Emitter

  • Scale Out
  • Does not scale up
    • Each instance will handle message
  • Do not dynamically create listeners
    • Memory Leaks
    • once instead of on

Queues

  • Design Pattern
    • Jobs are pushed onto a queue
    • Jobs processed once by a worker
    • First in first out (FIFO) order
    • Jobs are persistent
  • Redis Design
    • Locks and lists
    • Use a library, bull

Queues Example


var queue = require('bull');
var q = queue('simple queue');

q.process(function (job, done) {
  console.log(job.data);
  done();
});

q.add('a job');


a job

Queues

  • Does not scale out
    • Need a queue per application
  • Scales Up
  • Use case
    • Emails

Advanced Example

Conclusion

  • Cache
    • Share volatile data across instances and applications
    • Scales out
    • Scales up
  • Event Emitter
    • Share messages across multiple applications.
    • Scales Out
    • Does not scale up
  • Queue
    • Offload long running jobs onto scalable workers.
    • Does not scale out
    • Scales Up