Using webtask with ifttt to save reddit stories into mongodb

While reddit may not be the frontpage of internet for me, I do visit it often to keep in touch with programming, technology and sports news. Reddit has almost become my default bookmarking system where I often “save” reddit stories and read or refer them later. As the number of saved stories have gone up it has become quite difficult to search specific stories when I need them. There is no way to categorise them by subreddit either. I often thought about writing a node script or a cron job to fetch saved stories myself and dump them in a database or even a plaintext file for easy full-text search.

While there are ton of reddit related recipes on ifttt but none of them fit specifically my purpose. Then I was told about webtask.io and I immediately liked this idea of running a webtask, basically a piece of code that can be executed with a simple http call. Webtask combined with ifttt is a perfect combination as I get a ifttt trigger fired whenever I save a post on reddit and call a webtask action to do anything I want. I have complete control over the action part and not limited by the ifttt actions only.

So here is what I did briefly. I created a webtask that is invoked whenever I save a post on reddit. This webtask connects to mongolab mongodb instance and saves the reddit post meta data in “posts” collection. Later on I plan to write a node cli client to fetch/search these posts from terminal.

For now I will just document the webtask creation and ifttt configuration so just in case if you want to setup similar ifttt task you can follow along as well.

create webtask

A webtask is a simple node module. A simplest webtask could be as simple as a hello.js file containing following code

module.exports = function (cb) {
  cb(null, 'hello webtasks!');
}

and you can create a webtask using webtask cli (install it fist via https://webtask.io/cli)

wt create hello.js

Executing above command gives you a webtask url of sorts https://webtask.it.auth0.com/api/run/<some_magic_string>/hello?webtask_no_cache=1. To execute this webtask you simply have to call this url either via browser or via curl manually. Or with any http client library really. You can also host your webtask source code on a publically accessible url and create a webtask with that url.

For my purpose I created a github repo and pushed my redditwebtask into it. Now remember that webtask can be executed via http call so you can also pass some data to your webtask via query string. This is exactly what I needed. The idea is to use ifttt trigger to call redditwebtask via url and pass reddit post metadata to it in querystring. Let’s first take a look at redditwebtask implementation and see how it saves the metadata to mongolab mongodb instance.

var MongoClient = require('mongodb').MongoClient;

function savePost(post, db, cb) {
	var col = db.collection('posts');

	col.insertOne(post, function(err, r) {
		if(err) return cb(err);
    	console.log('inserted ', post);
    	cb(null);
	});
};

module.exports = function (ctx, done) { 
	MongoClient.connect(ctx.data.MONGO_URL, function (err, db) {
	    if(err) return done(err);

	    var post = {};
	    post.title = ctx.data.title;
	    post.content = ctx.data.content;
	    post.postURL = ctx.data.postURL;
	    post.subreddit = ctx.data.subreddit;

	    savePost(post, db, function(err) {
	    	if(err) return done(err);
	    	db.close();
	    	done(null, 'Success');
	    })
	});
};

As I said earlier webtask is a node module and behind the scenes it is executed inside a webtask runtime. This runtime provides generic and uniform execution environment which basically means access to lot of other popular node modules :)

Here I am “require”ing official mongodb node module to connect with mongodb instance.

The webtask module itself is a JavaScript function that takes two parameters; a ctx context object that provides access to any http querystring parameters passed to our webtask and a done callback function. There are different forms of this JavaScript function but this is one that is perfect for my use case.

Rest of the code is pretty self-explanatory. I connect to mongodb instance via MongoClinet.connect call with passed in MONGO_URI and save the post metadata in “posts” collection using mongodb driver api insertOne.

Next step is to actaully create our redditwebtask using wt-cli wt create command. Remember to create mongolab mongodb database and database user first as you need these to form your MONGO_URI connect parameter.

Creating webtask by executing wt create command return a url. Later you will use this url to setup an action in ifttt (step 6 below)

Create IFTTT trigger and action

Next, I added a recipe on ifttt.

Step 1. Choose reddit channel

Step 2. Choose “New post saved by you” trigger. This will take you to reddit login and authorize ifttt to certain permissions on your reddit account.

Step 3. Create trigger

Step 4. Next, choose Maker channel for the “that” action part in ifttt

Step 5. Select the “Make web request” action

Step 6. Configure ifttt to execute webtask by making a request to webtask url and pass “ingredients” i.e. reddit saved post metadata to the webtask

Troubleshooting

IFTTT takes bit of time to trigger the webtask web request. You can try the “Check recipe” buttom om ifttt once yur recipe is created to force the trigger.

If “Check recipe” throws any error, check the webtask logs by running wt logs command. This command streams the webtask logs so you might need to click “Check recipe” while this command is running as it won’t show past logs.

You could also test the webtask locally using the test.js in the repo (refer to readme in github repo to see how to use it).

Different ways of invoking JavaScript functions

In JavaScript the way a function is invoked has a significant impact on how the code within it executes, especially regarding this parameter.

If you have followed couple of my earlier posts I have demonstrated two ways of invoking functions already and they are

1. Invoking functions as, surprise, functions

2. Invoking functions as object methods

Now in addition to these two there are two more ways of invocations

3. Invoking functions as constructors

4. Invoking functions using apply() and call() methods

Before I dig into details of each type of function invocation, I must explain two important concepts i.e. two implicit parameters arguments and this that are passed to every invoked function.

arguments collection

Whenever a function is invoked in any of the four ways mentioned above, it can also be supplied with list of arguments which in turn are assigned to the parameters in the same order as defined in function declaration.

When the number of arguments match number of parameters then first argument is assigned to the first parameter, second argument is assigned to the second parameter and so on.

When the number of arguments are less than number of parameters then the parameters that have no corresponding arguments are set to undefined.

When the number of arguments are more than number of parameters then these extra arguments are simply not assigned to any parameters. Important thing to note though is that there is a way to access these extra arguments via implicit parameter called arguments

This implicit arguments parameter is quite deceptive. As in it acts like an array but it isn’t really an array. It has length property, its elements can be accessed using array notation and it can also be iterated just like an array can be. But it doesn’t have all array methods available on it.

this parameter

This notion of this implicit reference comes from object-oriented languages where methods are usually invoked in the context of an object and this generally points to this function context object. In JavaScript also if you invoke a function as an object’s method then its function context is the object and it means this implicitly references to the object. However invoking function as object method is just one of the four ways of invoking a function.

Let’s take a look at each of these four ways of invoking functions and what effect they have on the implicit this parameter they have in detail with sample codes

1. Invoking functions as functions

A function can be invoked simply using the () operator on any expression that resolves to a function reference.

function a() {
	console.log(this); // prints [object Window]
}
a();

When function is invoked in this manner the function context is Window i.e. the global context. This is true even when a function is deeply nested within other functions.

function a() {
    function b() {
       function c() {
          console.log(this); // prints [object Window]
       }
       c();
    }
    b();
}
a();

In above code, for function c function context is still Window

2. Invoking functions as object methods

As I described in earlier post, functions can be assigned to object properties and can be invoked using parentheses ().

Similar to object-oriented languages, if a function is invoked as an object’s method then the object becomes the function context.

var user = {'name' : 'Bon Jovi'};
user.sing = function() {
	console.log(this.name + ' is singing');
}
user.sing(); // prints Bon Jovi is singing

3. Invoking functions as constructors

Functions that can be invoked as constructors are just like any other functions. What makes a function a constructor function is the way it is invoked which is using new keyword.

When a function is invoked as a constructor function i.e. using new keyword,

1. A new empty Object is created and passed on to the function as this implicit parameter. this new Object becomes function context for the constructor function.

2. If nothing is returned from function, by default this newly created Object is returned from it.

function Singer() {
	this.sing = function() {
		console.log(this + ' is singing');
        return this;
    };
}
var user = new Singer();
console.log(user.sing() === user); // prints true

Note: Nothing is stopping from you to invoke a constructor function like a normal function i.e. without new keyword. In above example calling singer(); will get window object passed to it as its function context and get a sing method attached on it. Not exactly how you would want a constructor function to work.

4. Invoking function with apply() and call() methods

All the above ways of function invocation dictate what will be the function context But what if you want to force what will be the function context ? Well, function themselves provides you with the means to do so with apply() and call() methods. Wait a min though. Methods? methods of functions? Yes, you heard it right. Remember functions are first class objects and what do objects have? Objects have methods, and data of course. Every function is an object. Specifically, each function is an object of Function.

Ok, let’s see how to use apply() and call() methods to explicitely set this then.

var randomWeights = [10,20,30];

function addToBox() {
	var weight = 0;
	for (var i = 0; i < arguments.length; i++) {
		weight += arguments[i];
	}
	this.weight = weight;
}

var redBox = {};
var blueBox = {};

addToBox.apply(redBox, randomWeights);
addToBox.call(blueBox, 1,2,3,4,5);
console.log(redBox.weight); // prints '60'
console.log(blueBox.weight);// prints '15'

Both, apply() and call() methods takes first argument as the object that can be used as function context i.e. this. The difference between these two methods is how the rest of the arguments are supplied to the function. apply() takes an array of arguments whereas call() takes any number of arguments.

JavaScript scopes

I started my programming career with C,C++/VC++ and then Java. All are known as C family languages and they all share block level scope when it comes to variables and function declarations. In other words each block, typically represented by { and } braces, creates a new scope. For e.g.

#include <studio.h>

int main() {
	int a = 1;
	printf("%d ", a); //1

	if(1) {
		int a = 2;
		printf("%d ", a); //2
	}

	printf("%d ", a); //1
}

Scope of int a variable declared inside if block exists only till the end of if block.

However when I started working with JavaScript I tripped real hard understanding scopes in JavaScript. JavaScript’s C family like syntax had me fooled. The thing is JavaScript scopes are fundamentally different as they are defined by functions, not by blocks. For eg.

function a() {
	console.log(typeof b); // prints 'function'
	if(1) { 
		function b() {};
	}
	console.log(typeof b); // prints 'function'
}
a(); // prints 'undefined' as expected

Here function b is accessible i.e. prints ‘function’ even after the if block ends. In fact, function b can also be forward referenced before the if block. This is known as function hoisting.

Here comes the tricky part though. Variables declarations in JavaScript are in scope from their point of declaration inside a function till the end of function, irrespective of block nesting. For e.g.

function a() {
	console.log(typeof b); // prints 'undefined' because var b is not in scope yet
	if(1) { 
		var b = 1; // var b will be in scope till the end of function a
	}
	console.log(typeof b); // prints 'number'
}
a(); // prints 'undefined' as expected

Notice that unlike function b in previous sample code, var b here can not be forward referenced.

JavaScript functions travel first class

Functions in JavaScript are first class citizens of the language just like Objects.

In another words Functions enjoy all the same privileges (in fact more) as do the Objects in JavaScript.

Privileges such as

1. Functions can be created via a literal.

var obj = {a : 2, b: 2}; // object literal

var add = function(a, b) { // function literal
	return a + b;
};

2. Functions can be assigned to variables, array entries and object properties.

var add = function(a, b) { // function assigned to a variable
	return a + b;
};

var arr = [];
arr[0] = add; // function assigned to array entry at index 0

var obj = {};
obj.addMethod = add; // function assigned to Object obj as a addMethod property

3. Functions can be passed as arguments to other functions.

var values = [34, 56, 12, 11 , 5, 123];

var descendingComparator = function(value1, value2) {
	return value2 - value1;
}
values.sort(descendingComparator); // function descendingComparator passed as argument

console.log(values); // this should print [123, 56, 34, 12, 11, 5] 

4. Functions can be returned as values from other functions

function multiplier(x) {
	return function(y) { return x * y;};
}

var doubleIt = multiplier(2);
console.log(doubleIt(2)); // prints 4

var tripleIt = multiplier(3);
console.log(tripleIt(3)); // prints 9

5. Functions can also have their own properties and these properties can be added or removed dynamically

function myFunc() {}

myFunc.prop1 = 'prop1';

console.log(myFunc.prop1); // prints prop1

In addition to all of above, Functions also have special superpower in that they can be invoked

Prior writings

Prior to this blog I have written several articles & blogs on other sites. I just want to list them all here for easy access for myself later on.

List of articles in no particular order

• Explore the CDI programming model in ZK An article I wrote for IBM DeveloperWorks to introduce ZK CDI integration library that I had developed.

• Two part series of articles introducing how to integrate third party JavaScript libraries with ZK framework.

  • Part 1
  • Part 2

• MVVM in Java An article describing how to implement MVVM design pattern in Java using ZK framework.

• History management in ZK with HTML5 History API

• Securing ZK applications with Apache Shiro A How-To guide for securing ZK applications using Apache Shiro, a security framework.

• Three part article series introducing different ways of integrating MongoDB, a popular NoSQL database with ZK application

  • Part 1 - Using MongoDB Java Driver
  • Part 2 - Using Morphia
  • Part 3 - Using Spring Data

• ZK Spring Essentials Documentation I worked on ZK Spring integration library to upgrade it from Spring 2.x to Spring 3.x version and re-wrote this documentation to introduce various features.

• Absolutelayout component A blogpost introducing absolutelayout ZK component I had developed.

• Export ZK Spreadsheet to PDF A blogpost introducing a feature I built for ZK Spreadsheet, an excel like web component, that allowed exporting ZK Spreadsheet contents to a PDF for offline access or printing it.

• Handling ZK Events using CDI event notification model A blogpost I wrote to introduce ZK CDI integration library feature for using CDI event notification model.