A-D-D: I did a thing

(tldr; Repo: https://github.com/NorthMcCormick/A-D-D)

So I've been using Firebase religiously in my projects for the last year or two. It is so simple, which is awesome because my projects always need that kind of simplicity to excel.

Not saying that Firebase is missing anything but it doesn't have built-in denormalizing/aggregation with all of its SDKs, and if the SDK does have it, you have to build something to maintain it.

Over time I got into the habit of denormalizing my data on a Node server, it just made sense. Take the processing off the client and also make sure that it is happening correctly, right? This helped me strip down a lot of the code from the client and improve its snap. But for me, a truly lazy person, it was not enough.

That's where A-D-D comes in.

What is it

A-D-D (Automatic Data Denormalizer) is a library that works in your nodejs backend to provide some way of denormalizing data based off a pre-defined schema.

This means, you can maintain where your data goes, where it is updated, and where it is deleted, without having to write or wrap that code multiple times.

Jumping in

This isn't meant to be a full-blown tutorial, but rather a getting started guide.

If you're looking for more detailed docs, checkout the readme and examples in the repo: https://github.com/NorthMcCormick/A-D-D

Note: This library is in beta and is expected to morph and change dramatically until its 1.0 release. It is not recommended to use this in a production environment unless the specific beta release fits your needs, in which case fork it so I don't break your apps.

Core Concepts

Read through this, it will be confusing if you don't.

Database Handlers

A-D-D does not directly call your database, rather, you provide it a handler with a few pre-defined functions and let that do the magic. This allows you to have multiple databases running in parallel.

You could want this to duplicate some or all of your data to a second database for example.

There is a blank database handler template in the repo under databaseHandlers. There is also one for Firebase and one for IonicDB to get you started.

Note: Not all databases are created equal. When setting up your schema you need to be sure that you're not going out of the bounds of the database.

Example: With Firebase you can ask it to set a path like /users/all/abc/123/{{id}}, but IonicDB does not allow you to do that, the collections all sit at the same level so when writing the path in that schema you would want to make sure it's users/{{id}}.

Denormalizer

The denormalizer object is what you will mainly use. You create a denormalizer with a schema and then invoke the methods to create, update or remove data. The denormalizer will handle the rest from there.

A denormalizer corresponds to one set or object of data. In the example we provoke it with Tweets. If we wanted to denormalize a user's messages we would have another denormalizer object for that. Another for their profile information.

Schema

As mentioned earlier, this library uses a schema defined by you to know what data to use and where it should go.

Places

Each schema object has an array named places. A place is an object that holds the detail for where the data should go. You can have as many places as you want in a denormalizer with a minimum of 1.

Installation

It's just an NPM package. Install it with npm install a-d-d --save

Set up our databases

In this example we're going to set up two databases. One for Firebase and one for IonicDB. Go ahead and copy the database handlers into the root of your project (or wherever else you see fit) and require them where ever you are going to use the denormalizer.

Here's what mine looks like now:

var admin = require('firebase-admin');
var WebSocket = require('ws');
var IonicDB = require('@ionic/db').IonicDB

var serviceAccount = require('./firebase-creds.json');
var ionidbCreds = require('./ionicdb-creds.json');

var ADDConfig = require('./firebase-add/index.js').Config;
var ADDDenormalizer = require('./firebase-add/index.js').Denormalizer;

var ADDDatabase_Firebase = require('./add.firebase.db.js');
var ADDDatabase_IonicDB = require('./add.ionicdb.db.js');

Now we will create the database handlers and assign them to the config:


var firebaseAdminDB 	= admin.database(),
	myFirebaseDatabase 	= new ADDDatabase_Firebase(firebaseAdminDB);

var ionicAdminDb 		= new IonicDB(db_settings),
	myIonicDatabase  	 = new ADDDatabase_IonicDB(ionicAdminDb);

ADDConfig.database.ionic = myIonicDatabase;
ADDConfig.database.default = myFirebaseDatabase;

And that's it. Now we've got them all ready go to we can create our denormalizer.

Create your denormalizer

The schema object has some notes in the code, if you'd rather read about it in blog-form you can below the code sample.

var tweetDenormalizer = new ADDDenormalizer({
	schema: {
		expectingType: 'object',							// The type, could be number, string, object, array
		expectingProperties: ['handle', 'tweet'],			// The properties of the object (not required for other types, maybe)
		places: [{
			operation: 'set',								// Should we overwrite (set) or add to the list (push) these?
			type: 'object',									// What are we saving? If object, we expect 'properties' and if not we are just saving the value
			path: '/userTweets/{{userHandle}}/{{key}}',	// Where should we put this? 
			variables: {									// We can use variables in handlebars that map our input data to the path
				userHandle: 'handle',						// This will place our input handle to userHandle in the path
				key: '$key'
			},	
			properties: ['tweet']							// We only want to duplicate the tweet, not the handle over
		},
		{
			operation: 'push',								// Should we overwrite (set) or add to the list (push) these?
			type: 'string',									// What are we saving? If object, we expect 'properties' and if not we are just saving the value
			path: '/usersWhoTweeted',						// Where should we put this? 
			variables: {},
			property: 'handle'
		},
		{
			operation: 'set',								// Should we overwrite (set) or add to the list (push) these?
			type: 'string',									// What are we saving? A string, so it will just be a value in the database with a key
			path: '/lastUser',								// Where should we put this? 
			variables: {},
			property: 'handle',
			options: {
				ignore: {
					delete: true
				}
			}
		},
		{
			operation: 'set',
			type: 'object',
			path: 'allTweets/{{key}}',
			variables: {
				userHandle: 'handle',
				key: '$key'
			},	
			properties: ['tweet'],
			options: {
				database: 'ionic'
			}
		}]
	}
});

Okay there's a log going on here so I'll break it down.

Schema

The schema requires an input type (and if an object, a list of properties to be present). It will use this to filter out invalid requests so that you don't break your data.

Places

These are the objects describing how and where to put your data. Lets go through them one by one.

{
	operation: 'set',
	type: 'object',
	path: '/userTweets/{{userHandle}}/{{key}}',
	variables: {
		userHandle: 'handle',
		key: '$key'
	},		
	properties: ['tweet']
}

The operation tells A-D-D what to do with the data. Here we have set which will overwrite whatever data is in the path at that time. You can use push to add something to an array or collection too.

We define the type because that will change how it handles the data. Here we are using object because our input data looks like this (an object):

var tweetSample = {
	"handle": faker.name.firstName() + '_' + faker.name.lastName(),
	"tweet": "Wow hello this is my tweet, how cool is this"
};

The path is where we want this data to go. We did not define a database here so it will use the default which for us is Firebase.

You can include variables in your path. These variables will get their values from the input data. userHandle: 'handle', Tells A-D-D to create an internal variable named userHandle that is usable in the path and get its value from the handle property of the input object.

The last bit is the properties property. This allows you to select the top level properties of the input object that you want to duplicate. In this twitter example we only want the tweet we do not want the handle.

The next place:

{
	operation: 'push',
	type: 'string',
	path: '/usersWhoTweeted',
	variables: {},
	property: 'handle'
}

This one is noticeably shorter since we're not using any variables. We're just pushing the handle of the user as a string to the database.

What if we wanted to set who did something last? That's what our next place is:

{
	operation: 'set',
	type: 'string',
	path: '/lastUser',
	variables: {},
	property: 'handle',
	options: {
		ignore: {
			delete: true
		}
	}
}

It sets the handle of the last user who tweeted to that node. If the user deletes that tweet, we don't want it to remove the entire lastUser node so we can tell it to ignore that event. You can ignore update and delete events.

Last but not least we have a place in another database:

{
	operation: 'set',
	type: 'object',
	path: 'allTweets/{{key}}',
	variables: {
		userHandle: 'handle',
		key: '$key'
	},	
	properties: ['tweet'],
	options: {
		database: 'ionic'
	}
}

This is almost the same as the first place, but we are defining the database in the options.

Fire your denormalizer

Alrighty, we're so close! All we have to do is tell it to fire the events. The main function you will use is denormalize. It returns a promise with the success or failure if you need it and takes the object you want to denormalize.

var newTweet = {
	"handle": faker.name.firstName() + '_' + faker.name.lastName(),
	"tweet": "Wow hello this is my tweet, how cool is this"
};

tweetDenormalizer.denormalize(newTweet).then(function(result) {
	console.log('Denormalized', result);

}, function(error) {
	console.error('Could not denormalize', error);

}).catch(function(e) {
	console.log('Exception');
	console.log(e);
});

And that's basically it. You can put this in an event that happens when a tweet is received or if you're using this in a REST API fashion, in the api call itself.

Updates and deletes

Of course you want to keep your database clean and your data up to date. A-D-D can delete and update any data that was saved with set. Push is not supported yet.

With Firebase you can watch the children of a node and take actions based on if those children are changed or deleted. Here's how I would handle that:

firebaseAdminDB.ref('tweets').on('child_changed', function(snapshot) {
	var newTweet 		= snapshot.val();
		newTweet.$key 	= snapshot.key;

	tweetDenormalizer.update(newTweet).then(function(result) {
		console.log('Updated', result);

	}, function(error) {
		console.error('Could not update', error);

	}).catch(function(e) {
		console.log('Exception');
		console.log(e);
	});
});

firebaseAdminDB.ref('tweets').on('child_removed', function(snapshot) {
	var deletedTweet		= snapshot.val();
		deletedTweet.$key 	= snapshot.key;

	tweetDenormalizer.delete(deletedTweet).then(function(result) {
		console.log('Deleted', result);
	}, function(error) {
		console.error(error);
	}).catch(function(e) {
		console.error('Exception');
		console.error(e);
	})
});

Notice, we have to construct the object before sending it to the denormalizer, this is an example of how the database functionality is separated from the denormalizer itself.

Limitations

Right now because the library does not do any kind of data tracking or mapping, pushes are not able to be removed or updated via this library. For Firebase, as an example, without knowing or saving the key of the push we'd have to loop through the entire node to find which one it is and delete it.

This is something I'm working on but have not finalized a solution.

The other limitation is in types. Right now it officially supports objects and strings, but I'll be adding support for a more rigid validation of numbers, booleans, and other types.

Wanted features

If you're feeling adventurous I urge you to give it a try and see how you could use it in your projects and give me feedback on what you would like to see. There's a few specific items I have planned to implement already:

Complex objects

Right now, when defining object properties to copy or use you can only use the top level. While this works because you can modify the object before using it in the denormalizer to move whatever properties you need to the top, it's not ideal. The goal is to have a much more robust property handler throughout the lib.

Functions for data

The ability to define a function to manipulate data before sending it into each place. This might be a type conversion, validation, formatting, etc.

Events and hooks

You can use a customized database handler to fire events and kind of write hooks and callbacks but there will be more work done to make this a more integral part of the library.

I want your feedback

Please feel free to give suggestions and report issues in the repo or in the comments. I'm excited to see where this will go and I hope that someone else can find it useful.