Enterprise Integration Recipe: File Transfer Using Google Cloud Storage

This is one of a series of recipes on how to use products offered by Google Cloud Platform to implement enterprise integration solutions.

GCP Product used in this article:

• Google Cloud Storage,  
• Google Cloud Pub/Sub, 
• Google Cloud Functions

Implementation

System Diagram

Flow Chart for Subscriber

Recipe:

• gcs.tf: TerraForm code that provision and configure all GCP resource needed
• README.md: up to date instruction on how to use the recipe

Noteworthy Tips:

• GCS Object Lifecycle: make it easy to configure archive and error box. We should move old files to cold or nearline storage and eventually purge them. No more custom CRON jobs for this chore;
• Pubsub Notification for GCS: we configured notification for inbox "OBJECT_FINALIZE" event. Again no more CRON jobs needed to scan folders for new files. Another advantage is that the subscriber do not need to worry if we might be dealing with partially uploaded file;
• Stackdriver Monitoring or GCS: not configured in the recipe. But definitely worth exploring. For example, "object_count" might be used to monitor error box, and send alert when we see the count increasing too fast;
• Retry: this recipe implemented retry by simply do not "ACK" CPS messages. Cloud Pubsub will send notification again after ACK deadline expired for 7 days. So retry is implemented by CPS subscription's At-Least-Once Delivery feature. If want a shorter total retry period, you can add a line of code to check age of a message and mark it non-retriable if message age is over threshold. This approach saves the trouble of implementing reliably retry mechanism. Downside is lost control of retry intervals (for example, there is no exponential backoff retry)

"Permission denied to generate login hint for target domain."

Google OAuth Error Message: Permission denied to generate login hint for target domain.

Root Cause: development server, was using http://127.0.0.1:8080, even if I added it to cloud config, does not work. But once I change to http://localhost:8080, it starts to work.

Summer Fun with Google BigQuery

Nothing beats a game that can keep the whole family entertained for a few hours, while teaching kids some programming sense.

Just two weeks into summer, I already ran out of ideas to keep kids entertained.

Today, Google BigQuery come to the rescue. To be more precise, the USA Name Data provided hours of entertainment.

"This public dataset was created by the Social Security Administration and contains all names from Social Security card applications for births that occurred in the United States after 1879."

After showing my kids a few very simple queries of the USA Name Data, we started to play "Family Feud" game:

Most Popular Names

• Everyone write down 5 names in secret
• After we are done, each need to write their own query and run it in BQ to find out total count of these five names in all years
• Whoever gets the most counts wins
• Cannot reuse names in the following rounds

(After three rounds, we covered almost all the top 20 names in USA.)

Then we played another game with the dataset with these rules

Least Popular Names

• Write down 5 names in secret
• The names must draw at least one count
• Whoever gets the least counts wins

(We actually got one name with only 5 in all the last 103 years!)

Here is a sample query

SELECT SUM(number), name 
FROM [bigquery-public-data:usa_names.usa_1910_2013] 
WHERE name in ('John', 'Jason', 'Mary', 'David', 'William')
GROUP BY name

Now, after we are done with the game, the following two queries will show us the top answers.

Least Popular Names

SELECT SUM(number), name 
FROM [bigquery-public-data:usa_names.usa_1910_2013] 
GROUP BY name
ORDER BY 1

Most Popular Names

SELECT SUM(number), name 
FROM [bigquery-public-data:usa_names.usa_1910_2013] 
GROUP BY name
ORDER BY 1 DESC

Some Follow Up Questions

While the kids are still keen on it, some follow up questions will keep their brains from getting rusty:

• How many kids were born in the same year, and with the same name as yours?
• Which year has the most kids with the same name as yours?
• Are there any kids with your name in the opposite gender?
• Which years are "baby boom" years? (Hint: save to Google Sheet, and plot year vs. count(number))
• Which years have the most "Jacqueline"s and why? (Hint: opportunity for some history lessons about first ladies)
• Most gender neutral names (Hint: refer to this page)

Angular 2: Building a Google Map Component

Live Demo
Github Repo
 
In order to force myself dig deeper into Angular 2, I tried to build a component wrapper for Google Map JavaScript API. The end results is a demo component here: google_map.component.ts.

 

Sample Client Code

A sample client code using the component looks like this:

Features

The features including:

• Drawing through Angular binding
• Map events published as Angular events 

 

Tips

Here are some code snippets the highlights some of the lessons I have learned.

First, make sure Google Maps library is loaded from here:

    https://maps.googleapis.com/maps/api/js?key={YOUR_API_KEY}&libraries=visualization,place&callback=initMap

Add this line in typings.ts, so that we can have type definitions for Google Maps JavaScript API

    "googlemaps": "github:DefinitelyTyped/DefinitelyTyped/googlemaps/google.maps.d.ts"

Event handlers turns out to be tricky. My naive first implementation was something like this

this.map.addListener('click', function(e: google.maps.MouseEvent) {      
  this.mapClick.emit(e.latLng));
);

Turns out there are two big problems in this implementation:

1. this inside the event handler does not have member mapClick. Because it is no longer the component class
2. Once scope of this is fixed, I found data changes in the event handler does not trigger Angular change detection

Solutions for those problems? See the comments in my final implementation:

On the CSS side, it is very important to set height of Google Map component. If not done, it is automatically collapsed to nothing, and you won't see a map.

Demo 

Finally, a functional demo is here: google_map.html. Below is screenshot of the demo in action:

Angular 2: Injecting Parent and Child Component

Live Demo
Github Repo

To inject children component into a parent component, use ViewChildren (notice there is also ConentChildren which will inject children from content projection).

To inject a parent into child component, you simply need to inject parent in the constructor of child component. However, there is a bit of problem due to circular reference. The parent component has to reference child component as directive. If the child also depends on parent, we will get an error like this: EXCEPTION; Cannot resolve all parameters for '[ParentComponent]'.... So, the following slightly twisted injection is needed:

        contructor(@Inject(forwardRef(() => ParentComponent)) private _parent: ParentComponent)

A small working example showing both child and parent component injection can be downloaded here: https://github.com/huguogang/ng2tsdemo/blob/master/view_child.html.

Angular 2 Content, Content Projection, Transclusion

Live Demo
Github Repo

I guess under all these fancy words (especially Transclusion), there is a simple concept. And it seems to me, it is way easier to learn it by starting with looking at a working sample, make some changes and observe the changes.

I have created a bare minimum working example here: https://github.com/huguogang/ng2tsdemo/blob/master/content.html

Here are the key code fragments. Related elements are color coded.
Template for the directive:

<div>
  <h1><ng-content select="my-title"></ng-content></h1>
  <div>
    <ng-content select="content"></ng-content>
  </div>
  <br/>
  <em style="font-size:smaller"><ng-content select="footer"></ng-content></em>
</div>

Markup in the client component:

<multi-slot-content>
  <my-title>Angular 2 Content Demo</my-title>
  <content>The selectors defined in MultiSlotContent directive will find 
    the content element in this template, insert it into it's own template 
    section.</content>
  <footer>Footer: this feature was called transclusion in Angular 1.</footer>
</multi-slot-content>

Resulting HTML Markup:

<multi-slot-content>
  <div>
    <h1><my-title>Angular 2 Content Demo</my-title></h1>
    <div>
      <content>The selectors defined in MultiSlotContent directive will find 
        the content element in this template, insert it into it's own template 
        section.</content>
    </div>
    <br>
    <em style="font-size:smaller"><footer>Footer: this feature was called transclusion in Angular 1.</footer></em>
  </div>
</multi-slot-content>

Notice the key is to define proper Angular 2 selectors in the directive. Then in the client component, it just need to fill the content in the right selector target. The end result is the directive's ng-content elements got replaced by the fragments from client.

Angular 2 HTTP JSON Service

Live Demo
Github Repo

Here is a bare minimum demo of how to use Angular 2 HTTP JSON Service:https://github.com/huguogang/ng2tsdemo/blob/master/json_service.html

The demo uses observable instead of promise. A few potential catches:

• Import Rx.js in html file:

  <script src="node_modules/rxjs/bundles/Rx.js"></script>
  

• Import rxjs/Rx in service:

  import 'rxjs/Rx'

• Register HTTP_PROVIDERS

Angular 2 Renderer

Live Demo
Github Repo

Renderer in Angular 2 is a fascinating topic. It does not have an official documentation yet. As of this writing 5/1/2016, there is only a list of public method signatures here: https://angular.io/docs/ts/latest/api/core/Renderer-class.html.

It turns out it is very easy to get a hold of the renderer in your components. All it takes is to declare a Renderer in contructor, Angular 2 will inject it for you.

Code sample to inject renderer:

constructor(private _renderer: Renderer) { }

A working demo on github: https://github.com/huguogang/ng2tsdemo/blob/master/renderer.html

Angular 2 Resources

Official web site

https://angular.io

Official Github Repo

https://github.com/angular/angular
A few notable sub-directories:

• Source Code: https://github.com/angular/angular/tree/master/modules/angular2/src
• Release Progress: https://github.com/angular/angular/milestones

Angular CLI

https://github.com/angular/angular-cli
Command line interface. It helps to build skeleton code for a new project that follows best practice. Not officially released yet. But very helpful. It generates project with proper folder structure, unit testing and end2end tests are also included.

npm packages

• angular2: https://www.npmjs.com/package/angular2
  angular-cli: https://www.npmjs.com/package/angular-cli

Blog about Angular 2 (and other Software topics) by Victor Savkin

http://victorsavkin.com
There are not many tutorials here, but those few entries will give us a deep dive into Angular 2 design philosophy.

egghead.io

https://egghead.io/technologies/angular2
A series of short video tutorials about angular 2.

Stackoverflow tagged Angular 2

http://stackoverflow.com/questions/tagged/angular2