I make computers do thing that humans hate doing. I try to make them make things
193 stories
·
38 followers

Using Machine Learning to Explore Neural Network Architecture

1 Share


At Google, we have successfully applied deep learning models to many applications, from image recognition to speech recognition to machine translation. Typically, our machine learning models are painstakingly designed by a team of engineers and scientists. This process of manually designing machine learning models is difficult because the search space of all possible models can be combinatorially large — a typical 10-layer network can have ~1010 candidate networks! For this reason, the process of designing networks often takes a significant amount of time and experimentation by those with significant machine learning expertise.
Our GoogleNet architecture. Design of this network required many years of careful experimentation and refinement from initial versions of convolutional architectures.
To make this process of designing machine learning models much more accessible, we’ve been exploring ways to automate the design of machine learning models. Among many algorithms we’ve studied, evolutionary algorithms [1] and reinforcement learning algorithms [2] have shown great promise. But in this blog post, we’ll focus on our reinforcement learning approach and the early results we’ve gotten so far.

In our approach (which we call "AutoML"), a controller neural net can propose a “child” model architecture, which can then be trained and evaluated for quality on a particular task. That feedback is then used to inform the controller how to improve its proposals for the next round. We repeat this process thousands of times — generating new architectures, testing them, and giving that feedback to the controller to learn from. Eventually the controller learns to assign high probability to areas of architecture space that achieve better accuracy on a held-out validation dataset, and low probability to areas of architecture space that score poorly. Here’s what the process looks like:
We’ve applied this approach to two heavily benchmarked datasets in deep learning: image recognition with CIFAR-10 and language modeling with Penn Treebank. On both datasets, our approach can design models that achieve accuracies on par with state-of-art models designed by machine learning experts (including some on our own team!).

So, what kind of neural nets does it produce? Let’s take one example: a recurrent architecture that’s trained to predict the next word on the Penn Treebank dataset. On the left here is a neural net designed by human experts. On the right is a recurrent architecture created by our method:

The machine-chosen architecture does share some common features with the human design, such as using addition to combine input and previous hidden states. However, there are some notable new elements — for example, the machine-chosen architecture incorporates a multiplicative combination (the left-most blue node on the right diagram labeled “elem_mult”). This type of combination is not common for recurrent networks, perhaps because researchers see no obvious benefit for having it. Interestingly, a simpler form of this approach was recently suggested by human designers, who also argued that this multiplicative combination can actually alleviate gradient vanishing/exploding issues, suggesting that the machine-chosen architecture was able to discover a useful new neural net architecture.

This approach may also teach us something about why certain types of neural nets work so well. The architecture on the right here has many channels so that the gradient can flow backwards, which may help explain why LSTM RNNs work better than standard RNNs.

Going forward, we’ll work on careful analysis and testing of these machine-generated architectures to help refine our understanding of them. If we succeed, we think this can inspire new types of neural nets and make it possible for non-experts to create neural nets tailored to their particular needs, allowing machine learning to have a greater impact to everyone.

References

[1] Large-Scale Evolution of Image Classifiers, Esteban Real, Sherry Moore, Andrew Selle, Saurabh Saxena, Yutaka Leon Suematsu, Quoc Le, Alex Kurakin. International Conference on Machine Learning, 2017.

[2] Neural Architecture Search with Reinforcement Learning, Barret Zoph, Quoc V. Le. International Conference on Learning Representations, 2017.
Read the whole story
gms8994
173 days ago
reply
40291
Share this story
Delete

Automatic OCR with Hazel and PDFPen

1 Share

I have a useful scanner as part of my networked HP printer that will scan directly to a shared directory on my computer. Once there, I want the file to be renamed to the current date and the document OCR'd so that I can search it.

To do this, I use Hazel and PDFPen and this is a note to ensure that I can remember to do it again if I ever need to!

Firstly, rename the file. My scanner names each file with the prefix scan, so the Hazel rule is quite simple:

If all the following conditions are met:
	Name starts with scan

Do the following to the matched file or folder:
	Rename with pattern: [date created][extension]

This is the screenshot:

Hazel1

Having renamed the file, we can use PDFPen's AppleScript support to perform an OCR of the document:

If all the following conditions are met:
	Extension is pdf
	Date Last Modified is after Date Last Matched

Do the following to the matched file or folder:
	Run AppleScript embedded script

The embedded AppleScript is:

tell application "PDFpen"
	open theFile as alias
	tell document 1
		ocr
		repeat while performing ocr
			delay 1
		end repeat
		delay 1
		close with saving
	end tell
	quit
end tell

This is the screenshot of it in Hazel:

Hazel2

That's it. Scanning a document now results in a dated, OCR'd PDF file in my Scans folder.

Read the whole story
gms8994
314 days ago
reply
40291
Share this story
Delete

Jeff Bezos assures employees that HR working 100 hours a week to address their complaints

1 Comment
I see what you did there!∞ Read this on The Loop
Read the whole story
gms8994
819 days ago
reply
God damnit... I got hooked by The Onion again!
40291
Share this story
Delete

Profiling Ag. Writing My Own Scandir

1 Share

Although I benchmarked every revision of Ag, I didn’t profile them all. After looking at the graph in my previous post, I profiled some of the revisions where performance changed significantly.

This is a run of revision a87aa8f8; right before I reverted the performance regression. You can see it spends 80% of execution time in fnmatch().

This is tagged release 0.9. Much faster, and it only spends about half the time in fnmatch().

Finally, here’s a run after merging pull request #56. This fixed issue #43 and improved performance for many cases. I’m rather proud of that pull request, since it fixed a lot of issues. The rest of this post explains the specific changes I made to get everything working the way I wanted.

To start with, I should explain Ag’s old behavior. Before I merged that pull request, Ag called scandir() on each directory. Then scandir() called filename_filter() on every entry in the directory. To figure out if a file should be ignored, filename_filter() called fnmatch() on every entry in the global char *ignore_patterns[]. This set-up had several problems:

  1. scandir() didn’t let me pass any useful state to filename_filter(). The filter could only base its decision on the dirent and any globals.
  2. ignore_patterns was just an array of strings. It couldn’t keep track of a hierarchy of ignore files in subdirectories. This made some ignore entries behave incorrectly (issue #43). This also hurt performance.

Fixing these issues required rejiggering some things. First, I wrote my own scandir(). The most important difference is that my version lets you pass a pointer to the filter function. This pointer could be to say… a struct containing a hierarchy of ignore patterns.

Surprise surprise, the next thing I did was make a struct for ignore patterns:

struct ignores {
    char **names; /* Non-regex ignore lines. Sorted so we can binary search them. */
    size_t names_len;
    char **regexes; /* For patterns that need fnmatch */
    size_t regexes_len;
    struct ignores *parent;
};

This is sort of an unusual structure. Parents don’t have pointers to their children, but they don’t need to. I simply allocate the ignore struct, search the directory, then free the struct. This is done around line 340 of search.c. Searching is recursive, so children are freed before their parents.

The final change was to rewrite filename_filter(). It calls fnmatch() on every entry in the ignore struct passed to it. If none of those match and ig->parent isn’t NULL, it repeats the process with the parent ignore struct, and so-on until it reaches the top.

All-in-all, not a bad change-set. I fixed a lot of things I’d been meaning to fix for a while. I also managed to clean up quite a bit of code. If not for my re-implementation of scandir(), the pull request would have removed more lines than it added.

One last thing: I’d like to praise a piece of software and criticize another. I tip my hat to Instruments.app. I’ve found it invaluable for finding the causes of many memory leaks and performance issues. But I wag my finger at git. Git allows .gitignore files in any directory, and it allows these files to contain regular expressions. Worse, these regexes can reference sub-directories. For example, foo/*/bar is a valid ignore pattern. Regular expressions plus directory hierarchies translate to complicated implementations and confusing behavior for users. It’s no fun for anyone involved.

Go dark.

Read the whole story
gms8994
823 days ago
reply
40291
Share this story
Delete

Two Fucking Developers - Not just another blog about coding

1 Share

This series of AngularJS best practices articles are defined by three rules: it’s easier to debug, it’s easier to test and it’s easier to maintain. So maybe this could be worst for other things (like performance).

$routeProvider

AngularJS $routeProvider is probably one of the most popular and widely used providers. With $routeProvider we manage all our application routes and is responsible for updating our views and pairing these views with its controllers.

If you have been worked with AngularJS for a while is probably that you are familiar with $routeProvider and this could be an approach of how you’re using it currently:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

"use strict";

angular.module('httpExample', ["ngRoute"])

    .config(["$routeProvider", function($routeProvider) {

        $routeProvider

            .when("/", {

                templateUrl: "main.html",

                controller: "MainCtrl",

                resolve: {

                    myData: ["$http", function($http){

                        return $http({url: "data.json"});

                    }]

                }

            })

    }])

    .controller("MainCtrl", ["$scope", "myData", function ($scope, myData) {

        $scope.data = myData.data;

    }]);

This is a classic example of AngularJS $routeProvider usage. You have your view in a separated html and your controller takes your data from a resolve function that is loading an external json. Everything looks fine, but as you will see, this could be improved to have something better to scale, maintain and test.

Using $routeProvider

The biggest mistake in the previous approach is that our data source is completely attached to our $routeProvider and thats wrong because we can’t reuse it. Also there’s another problem, it is going to be harder to test and also we don’t have a centralized service to scale our data provider. So our first improvement would be to create a service to load our data.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

"use strict";

angular.module('httpExample', ["ngRoute"])

    .config(["$routeProvider", function($routeProvider) {

        $routeProvider

            .when("/", {

                templateUrl: "main.html",

                controller: "MainCtrl",

                resolve: {

                    myData: ["httpDataLoader", function(httpDataLoader){

                        return httpDataLoader.load();

                    }]

                }

            })

    }])

    .controller("MainCtrl", ["$scope", "myData", function ($scope, myData) {

        $scope.data = myData.data;

    }])

    .service("httpDataLoader", ["$http", function($http) {

        this.load = function() {

          return $http({url: "data.json"});

        }

    }]);

That’s much better! Now we have an injectable service called “httpDataLoader” that we can use along our application and also we can scale it. If you’re working with a REST service you can even use Restangular, an awesome module created by Martin Gontovnikas. This is how your service should looks like with Restangular:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

"use strict";

angular.module('httpExample', ["ngRoute", "restangular"])

    .config(["$routeProvider", function($routeProvider) {

        $routeProvider

            .when("/", {

                templateUrl: "main.html",

                controller: "MainCtrl",

                resolve: {

                    myData: ["httpDataLoader", function(httpDataLoader){

                        return httpDataLoader.load();

                    }]

                }

            })

    }])

    .controller("MainCtrl", ["$scope", "myData", function ($scope, myData) {

        $scope.data = myData.data;

    }])

    .service("restService", ["Restangular", function(Restangular) {

        this.getUsers = function() {

          return Restangular.all("users").getList();

        }

    }]);

Obviously to have a provider for our data will allow us to test it better by mocking our service, so as we’ve seen this is a highly recommendable technique and your data should always come from a service.

Now that we have been solved our biggest problem let’s see what else we can do.

Simplifying scope with “Controller as”

Working with scopes in JavaScript sucks. I suffered scopes back in the old times with Flash 5/ActionScript 1.0 and I can’t believe that today we’re still seeing things like “this._self”.

Anyway, AngularJS manages quite good scopes for us, however there is still a common problem with “scopes” and “this”. Let’s see our controller again:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

"use strict";

angular.module('httpExample', ["ngRoute"])

    .config(["$routeProvider", function($routeProvider) {

        $routeProvider

            .when("/", {

                templateUrl: "main.html",

                controller: "MainCtrl",

                resolve: {

                    myData: ["httpDataLoader", function(httpDataLoader){

                        return httpDataLoader.load();

                    }]

                }

            })

    }])

    .controller("MainCtrl", ["$scope", "myData", function ($scope, myData) {

        var defaultMaxItems = 5;

        $scope.data = myData.data;

        if ($scope.data.maxItems) defaultMaxItems = $scope.data.maxItems;

    }])

    .service("httpDataLoader", ["$http", function($http) {

        this.load = function() {

          return $http({url: "data.json"});

        }

    }]);

Ok, now we’ve been added a new variable called “defaultMaxItems” that is overriden by our data provider. However we have a problem here because we’ve been defined “defaultMaxItems” as a “private” variable and now it is not available in scope and it is not testable or accesible for our html template. So let’s fix this with a common workaround:

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

27

28

29

"use strict";

angular.module('httpExample', ["ngRoute"])

    .config(["$routeProvider", function($routeProvider) {

        $routeProvider

            .when("/", {

                templateUrl: "main.html",

                controller: "MainCtrl",

                resolve: {

                    myData: ["httpDataLoader", function(httpDataLoader){

                        return httpDataLoader.load();

                    }]

                }

            })

    }])

    .controller("MainCtrl", ["$scope", "myData", function ($scope, myData) {

        $scope.controller = this;

        var defaultMaxItems = 5;

        var data = myData.data;

        if (data.maxItems) defaultMaxItems = data.maxItems;

    }])

    .service("httpDataLoader", ["$http", function($http) {

        this.load = function() {

          return $http({url: "data.json"});

        }

    }]);

Ok, that’s better. Now we have all defined in “this” and it is accesible through scope to test and bind with the template. Basically, the AngularJS team arrived to this conclusion and since AngularJS 1.1.5 provide a way to do this automatically with “Controller as”. If you want to know more you can read this post where Alex Ford explains perfectly how “Controller as” works.

1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

21

22

23

24

25

26

"use strict";

angular.module('httpExample', ["ngRoute"])

    .config(["$routeProvider", function($routeProvider) {

        $routeProvider

            .when("/", {

                templateUrl: "main.html",

                controller: "MainCtrl",

                controllerAs: "controller",

                resolve: {

                    myData: ["httpDataLoader", function(httpDataLoader){

                        return httpDataLoader.load();

                    }]

                }

            })

    }])

    .controller("MainCtrl", ["myData", function (myData) {

        this.data = myData.data;

    }])

    .service("httpDataLoader", ["$http", function($http) {

        this.load = function() {

          return $http({url: "data.json"});

        }

    }]);

The first thing that you will notice is that with “Controller as” you no longer need to inject $scope, “Controller as” will do that for you and will create also a variable called “controller” inside “$scope” that will be “this” ($scope.controller = this;). This is so much clean and easy to use than injecting $scope and matching to this, etc.

With this example I’ve created a Plunk just in case you want to play with the code.

Happy fuckingn coding!

Read the whole story
gms8994
826 days ago
reply
40291
Share this story
Delete

08/06/14 PHD comic: 'Do you have a minute?'

5 Comments and 19 Shares
Piled Higher & Deeper by Jorge Cham
www.phdcomics.com
Click on the title below to read the comic
title: "Do you have a minute?" - originally published 8/6/2014

For the latest news in PHD Comics, CLICK HERE!

Read the whole story
gms8994
1199 days ago
reply
40291
popular
1201 days ago
reply
Share this story
Delete
4 public comments
emdeesee
1198 days ago
reply
Yeah, I call it "working" too.
Lincoln, NE
iaravps
1200 days ago
reply
Story of my life
Rio de Janeiro, Brasil
expatpaul
1201 days ago
reply
Too true, for all of us
Belgium
Next Page of Stories