Peterbe.com

MozTrap is what's called a "test case management system". Basically, software QA people need a structure and pattern to their testing. What to test, what versions to test on and what hardware/operatting system etc all is part of a "test suite". That's what MozTrap manages.

So this project was built by Mozilla's automation and tools team. It is currently not an actively developed project. Not because it's not needed or used but because it basically maps all the features we need. A large part of the code base was originally written by a personal friend of mine who I respect wholeheartedly; Carl Meyer of Django/pip/virtualenv/etc fame. I'm grateful for the awesome documentation he left behind amongst many other things.

Together with the team we sat down and listed all the biggest pain points as of today. Basically, the number one thing is speed. Pages load too slowly. Normally when web developers worry themselves with web performance it's a matter of shaving milliseconds off a page where a clients perception equals lost or gained profits. Here's not a problem of milliseconds but a problem of seconds. After some quick poking around on the production site and looking at some code the conclusion is simple: The site is so darn slow because the HTML sent from the server is way too MASSIVE. And baked into that is a mixture of the poor web server having to produce a massive HTML blob and it being sent over the wire.

One test run I made said it took 14 seconds to render a certain page.

Why is it so slow?

So how did this happen and why is it not Carl's fault? :) The reason it happened was because of the underestimated number of options added to the advanced filtering drop-downs. On a local dev version you never notice these things because you set up some options, for example tags, and the drop-down never gets larger than 10-20 options. For example, the "Creator" drop-down today has 1,664 different choices.

If you take all those choices and turn thing into a HTML like this: <option value="1">Adam</option>\n<option value="2">Bram</option>... etc. you get 66Kb of just HTML. However, MozTrap doesn't work like that. Instead it uses pretty drop-downs that don't look like regular HTML drop-downs. See for yourself; go to https://moztrap.mozilla.org/results/runs/ and click the "Advanced Filtering" button.
So, that means that the HTML for each option instead looks like this:

<li class="filter-item">
  <input name="filter-creator" data-name="creator" value="1" id="id-filter-creator-1" class="check" type="checkbox">
  <span class="onoff">
    <label for="id-filter-creator-1" class="onoffswitch">Adam</label>
                <span class="pinswitch"></span>
    <span class="content" title="creator: Adam">Adam</span>
  </span>
</li>

Now you get 620Kb of just HTML just for the "Creators" field. Granted, that is the biggest field of all the drop-downs but lots of them are massive.

So, this makes the page weigh a total of about 1.1Mb just for the HTML. Not only is it a lot of work for the (Django) server to generate this but it's also a heck of a lot of data to send across the Internet on every page request.

So what was the solution?

An ideal solution would have been a significant re-write whereby much of the values of the page gets rewritten as later AJAX calls. I.e. load a skeleton that loads superfast, and then load some AJAX in the background. That AJAX could potentially be cached in the browser with localStorage or something so that you get something to show very quickly whilst you wait for the AJAX request to complete. But this would have been too big a change and the way the filtering works on these pages, you actually need all the options in the drop-downs on immediate load because of the way "pinned filters" work.

So the solution was to replace all the repeated HTML chunks with 1 JSON string and then a piece of Javascript template rendering. So, in the Django template code instead of:

{% for field in filters %}
  {% include "lists/_filter_group.html" with advanced=1 prefix="filter" pinable=1 %}
{% endfor %}

We now replace this with:

<script>
var FILTERS = {% filterset_to_json filters with advanced=1 prefix="filter" pinable=1 %}
</script>
<script id="filter_group" type="text/html">
<section class="filter-group {{ field.cls }}" data-name="{{ field.key }}">
  <h5 class="category-title">
    {{ _field_name_lower }}
    {{# field.switchable }}
    ...

What that basically is is some Mustache code that I use to generate the HTML DOM nodes and insert into the page after load.

In conclusion

So basically nothing changes. Nothing of the Django view had to change. Visually there's no difference and the same actual user data is sent from the server to the client but just packed in a more optimal way.

There are multiple pages where these massive "Advanced Filtering" options exist but on one page I measured the whole page went from weighing 1.1Mb down to 132Kb.

On Friday I did a Show HN and got featured on the front page for HTML Tree.

Amazingly, out of the 3,858 visitors (according to Google Analytics today) 2,034 URLs were submitted and tested on the app. Clearly a lot of people just clicked the example submission but out of those 1,634 were unique. Granted, some people submitted more than one URL but I think a large majority of people came up with a URL of their own to try. Isn't that amazing! What a turnout of a Friday afternoon hack (with some Sunday night hacking to make it into a decent looking website).

The lesson to learn here is that the Hacker News crowd is excellent for getting engagement. Yes, there are a lot of blather and almost repetitive submissions but by and large it's a very engaging community. Suck on that those who make fun of HN!

UPDATE

HTMLTree has been moved to https://htmltree.herokuapp.com/

If you run Homebrew on your OSX and you use that to install postgres you will have noted there's a new formula for Postgres 9.3(.4). Yay! (actually this was done many months ago but I'm slow to keep my brews upgraded)

When you run the upgrade you'll notice that psql no longer works because the server can't start.

Bummer! But there's hope: This excellent blog post

That's all you need. ...unless you have some uncompatible library extension installed. E.g. json_enhancements

The problem is that you can't install json_enhancements into a Postgres 9.3 server (json_enhancements is a backport from 9.3 to desperate people still on 9.2). And you can't do the upgrade because the new server has one less installed library. You'll get this after a failing pg_upgrade:

peterbe@mbp:~$ cat loadable_libraries.txt
Could not load library "$libdir/json_enhancements"
ERROR:  could not access file "$libdir/json_enhancements": No such file or directory

I left some more details about this on the pgsql-admin mailing list.

The trick that worked for me was to start the old 9.2 server like this:

/usr/local/Cellar/postgresql/9.2.2/bin/postgres -D /usr/local/var/postgres -p 9000

And now I can open psql against the old data[base] and drop the extension:

$ psql -p 9000 mydatabase
psql (9.3.4, server 9.2.2)
mydatabase=# drop extension json_enhancements;

After I had done that I was able to successfully complete the pg_upgrade.

I hope this helps some other poor sucker stuck in the same chicken and egg situation.

UPDATE

A much better blog post to guide you is Ketia's Blog. For example, here's a post about upgrading from 9.4 to 9.5.

angular-classy, by @DaveJ, is an interesting AngularJS module that you use to get some class structure into your controller. You can check out his page and the documentation there for some basic examples of that it does.

This appeals to me as a Python developer because now my angular code looks more like a Python class. I also like that there's an init function now (similar to python's __init__ I guess) and I also like that you can distinguish between "scope functions" and "local functions". To explain that, consider this:

  // somewhere in a controller 
  ...


  $scope.addSomething = function() {  // used in your template
    if ($scope.some_precondition) {
      reallyAddSomething($scope.firt_name, $scope.last_name);
    }
  };

  function reallyAddSomething(first_name, last_name) {
    // can still use $scope in here
  }

And compare this with angular-classy:

  // somewhere in a controller 
  ...


  addSomething: function() {  // used in your template
    if (this.$scope.some_precondition) {
      this._reallyAddSomething(this.$scope.first_name, this.$scope.last_name);
    }
  },

  _reallyAddSomething = function(first_name, last_name) {
    // can still use this.$scope in here
  },

Basically, the _ prefix makes the function available on this but not attached to the scope. And I think that just makes sense!

So my guttural feeling is all positive about angular-classy. But there is still one big caveat. The mythical "this" in Javascript. It's great but it's kinda clunky too because it rebinds in every sub-scope. The solution to that is to bind things. For example, for a success promise it now has to look like this:

this.$http.get('/some/url')
.success(function(response) {
  this.somethingElseInTheModule(response.something);
}.bind(this));

Anyway, let's compare the before and after of a real project.

Before

controllers.js

After

controllers.js

What do you think? Does it look better? Full diff here

I think I like it. But I need to let it "sink in" a bit first. I think the code looks neater with angular-classy but it's now a new dependency and it means that people who know angular but not familiar with angular-classy would get confused when they are confronted with this code.

UPDATE

I merged the branch. So now this project is classy.

I have now closed issue #2 on github-pr-triage. So, now you can have a dashboad of every GitHub project whose pull requests you care about.

The only format of using just 1 repo works too. E.g. /owner/project) and should hopefully not break anybody's bookmarks. The new format for having multiple repos across (possibly) multiple owners is like this:

owner1:projectA,projectB;owner2:projectX,projectY,projectZ

See screenshot:

To set yours up, here's a running instance available on https://prs.paas.allizom.org

Last weekend I installed a Wattvision ("real-time energy monitoring sensors") in my house. It's so you can measure how much electricty your house is using. In real-time.

So it comes in two parts:

1) A camera sensor that is attached to the electricty meter. It stares at the rotating disk all day.

2) A little router/sensor thing that is connected to the camera and connects, by Wi-Fi, to your home router.

Then, the little router/sensor sends all your measurments to wattvision.com's servers. After that, I sign in to Wattvision (using my Google account) and there I can get all the statitics about my house electricity. Simple, ah?

Wattvision started as a Kickstarter project two years ago and since I sponsorered that project they sent me a kit now that it's fully tested and working. Yay!

The installation was almost jokingly simple to set up! It had that lovely "just works" feeling to it. The only challenging part was to pull the sensor wire from the corner of the house to a good spot in our basement. My wife, who is much shorter than me, crawled under our crawl-space and helped me hook it all up. I was just so impressed with the instructions. They were very well written.

Now that it's set up, you get all your statistics and graphs by signing in to wattvision.com and it works great on mobile as well. I have to admit, at this point, I really haven't understood what it all does and what it all means. Besides, because I only installed it a week ago, I don't yet have enough data to compare current usage with historic usage. By the way, you can download your data in CSV form too.

The sexiest feature is to be able to sit and watch your graph and then you deliberately switch something on in the house and you can see the graph "spike". Obviously the height of the spike depends on what you're switching on. For example, an LED light I don't even think it registers (admittedly, haven't tested that yet).

I think this is the key reason to have Wattvision; to get an insight into what in your household causes the most energy consumption. Having said that, we're not going to stop taking showers.

In conclusions...

You simply can't have data analysis without data collection. Also, if there's anything you want to trim, such as body fat, awareness is usually a very good weapon.

I don't know if I'll be checking back into the statistics very often. The novelty might just wear off after a while. We'll see.

grymt is a python tool that takes a directory full of .html, .css and .js and prepares the html for optimial production use.

For a teaser:

1. Look at the "input"

2. Look at the "output" (Note! You have to right-click and view source)

So why did I write my own tool and not use Grunt?!

Glad you asked! The reason is simple: I couldn't get Grunt to work.

Grunt is a framework. It's a place where you say which "recipes" to execute and how. It's effectively a common config framework. Like make.
However, I tried to set up a bunch of recipes in my Gruntfile.js and most of them worked well individually but it was a hellish nightmare to get it all to work together just the way I want it.

For example, the grunt-contrib-uglify is fine for doing the minification but it doesn't work with concatenation and it doesn't deal with taking one input file and outputting to a different file.
Basically, I spent two evenings getting things to work but I could never get exactly what I wanted. So I wrote my own and because I'm quite familiar with this kind of stuff, I did it in Python. Not because it's better than Node but just because I had it near by and was able to quicker build something.

So what sweet features do you get out of grymt?

1. You can easily make an output file have a hash in the filename. E.g. vendor-$hash.min.js becomes vendor-64f7425.min.js and thus the filename is always unique but doesn't change in between deployments unless you change the files.

2. It automatically notices which files already have been minified. E.g. no need to minify somelib.min.js but do minify otherlib.js.

3. You can put $git_revision anywhere in your HTML and this gets expanded automatically. For example, view the source of buggy.peterbe.com and look at the first 20 lines.

4. Images inside CSS get rewritten to have unique names (based on files' modified time) so they can be far-future cached aggresively too.

5. You never have to write down any lists of file names in soome Gruntfile.js equivalent file

6. It copies ALL files from a source directory. This is important in case you have something like this inside your javascript code: $('<img>').attr('src', 'picture.jpg') for example.

7. You can chose to inline all the minified and concatenated CSS or javascript. Inlining CSS is neat for single page apps where you have a majority of primed cache hits. Instead of one .html and one .css you get just one .html and the amount of bytes is the same. Not having to do another HTTP request can save a lot of time on web performance.

8. The generated (aka. "dist" directory) contains everything you need. It does not refer back to the source directory in any way. This means you can set up your apache/nginx to point directly at the root of your "dist" directory.

So what's the catch?

1. It's not Grunt. It's not a framework. It does only what it does and if you want it to do more you have to work on grymt itself.

2. The files you want to analyze, process and output all have to be in a sub directory.
   Look at how I've laid out the files here in this project for example. ALL files that you need is all in one sub-directory called app. So, to run grymt I simply run: grymt app.

3. The HTML files you throw into it have to be plain HTML files. No templates for server-side code.

How do you use it?

pip install grymt

Then you need a directory it can process, e.g ./client/ (assumed to contain a .html file(s)).

grymt ./client

For more options, check out

grymt --help

What's in the future of grymt?

If people like it and want to add features, I'm more than happy to accept pull requests. Some future potential feature work:

• I haven't needed it immediately, yet, myself, but it would be nice to add things like coffeescript, less, sass etc into pre-processing hooks.

• It would be easy to automatically generate and insert a reference to a appcache manifest. Since every file used and mentioned is noticed, we could very accurately generate an appcache file that is less prone to human error.

• Spitting out some stats about number bytes saved and number of files reduced.

My friend and colleague Jannis (aka jezdez) Leidel saved my bacon today where I had gotten completely stuck.

So, I have this python2.6 virtualenv and whenever I ran python setup.py sdist upload it would upload a really nasty tarball to PyPI. What would happen is that when people do pip install premailer it would file horribly and look something like this:

...
IOError: [Errno 2] No such file or directory: '/path/to/virtual-env/build/premailer/setup.py'

What?!?! If you download the tarball and unpack it you'll see that there definitely is a setup.py file in there.

Anyway. What happens, which I didn't realize was that within the .tar.gz file there were these strange copies of files. For example for every file.py there was a ._file.py etc.

Here's what the file looked like after a tarball had been created:

(premailer26)peterbe@mpb:~/dev/PYTHON/premailer (master)$ tar -zvtf dist/premailer-2.0.2.tar.gz
-rwxr-xr-x  0 peterbe staff     311 Apr 11 15:51 ./._premailer-2.0.2
drwxr-xr-x  0 peterbe staff       0 Apr 11 15:51 premailer-2.0.2/
-rw-r--r--  0 peterbe staff     280 Mar 28 10:13 premailer-2.0.2/._LICENSE
-rw-r--r--  0 peterbe staff    1517 Mar 28 10:13 premailer-2.0.2/LICENSE
-rw-r--r--  0 peterbe staff     280 Apr  9 21:10 premailer-2.0.2/._MANIFEST.in
-rw-r--r--  0 peterbe staff      34 Apr  9 21:10 premailer-2.0.2/MANIFEST.in
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/._PKG-INFO
-rw-r--r--  0 peterbe staff    7226 Apr 11 15:51 premailer-2.0.2/PKG-INFO
-rwxr-xr-x  0 peterbe staff     311 Apr 11 15:51 premailer-2.0.2/._premailer
drwxr-xr-x  0 peterbe staff       0 Apr 11 15:51 premailer-2.0.2/premailer/
-rwxr-xr-x  0 peterbe staff     311 Apr 11 15:51 premailer-2.0.2/._premailer.egg-info
drwxr-xr-x  0 peterbe staff       0 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/
-rw-r--r--  0 peterbe staff     280 Mar 28 10:13 premailer-2.0.2/._README.md
-rw-r--r--  0 peterbe staff    5185 Mar 28 10:13 premailer-2.0.2/README.md
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/._setup.cfg
-rw-r--r--  0 peterbe staff      59 Apr 11 15:51 premailer-2.0.2/setup.cfg
-rw-r--r--  0 peterbe staff     280 Apr  9 21:09 premailer-2.0.2/._setup.py
-rw-r--r--  0 peterbe staff    2079 Apr  9 21:09 premailer-2.0.2/setup.py
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._dependency_links.txt
-rw-r--r--  0 peterbe staff       1 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/dependency_links.txt
-rw-r--r--  0 peterbe staff     280 Apr  9 21:04 premailer-2.0.2/premailer.egg-info/._not-zip-safe
-rw-r--r--  0 peterbe staff       1 Apr  9 21:04 premailer-2.0.2/premailer.egg-info/not-zip-safe
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._PKG-INFO
-rw-r--r--  0 peterbe staff    7226 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/PKG-INFO
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._requires.txt
-rw-r--r--  0 peterbe staff      23 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/requires.txt
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._SOURCES.txt
-rw-r--r--  0 peterbe staff     329 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/SOURCES.txt
-rw-r--r--  0 peterbe staff     280 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/._top_level.txt
-rw-r--r--  0 peterbe staff      10 Apr 11 15:51 premailer-2.0.2/premailer.egg-info/top_level.txt
-rw-r--r--  0 peterbe staff     280 Apr  9 21:21 premailer-2.0.2/premailer/.___init__.py
-rw-r--r--  0 peterbe staff      66 Apr  9 21:21 premailer-2.0.2/premailer/__init__.py
-rw-r--r--  0 peterbe staff     280 Apr  9 09:23 premailer-2.0.2/premailer/.___main__.py
-rw-r--r--  0 peterbe staff    3315 Apr  9 09:23 premailer-2.0.2/premailer/__main__.py
-rw-r--r--  0 peterbe staff     280 Apr  8 16:22 premailer-2.0.2/premailer/._premailer.py
-rw-r--r--  0 peterbe staff   15368 Apr  8 16:22 premailer-2.0.2/premailer/premailer.py
-rw-r--r--  0 peterbe staff     280 Apr  8 16:22 premailer-2.0.2/premailer/._test_premailer.py
-rw-r--r--  0 peterbe staff   37184 Apr  8 16:22 premailer-2.0.2/premailer/test_premailer.py

Strangly, this only happened in a Python 2.6 environment. The problem went away when I created a brand new Python 2.7 enviroment with the latest setuptools.

So basically, the fault lies with OSX and a strange interaction between OSX and tar.
This superuser.com answer does a much better job explaining this "flaw".

So, the solution to the problem is to create the distribution like this instead:

$ COPYFILE_DISABLE=true python setup.py sdist

If you do that, you get a healthy lookin tarball that actually works to pip install. Thanks jezdez for pointing that out!

Buggy is a singe-page webapp that relies entirely on the Bugzilla Native REST API. And it works offline. Sort of. I say "sort of" because obviously without a network connection you're bound to have outdated information from the bugzilla database but at least you'll have what you had when you went offline.

When you post a comment from Buggy, the posted comment is added to an internal sync queue and if you're online it immediately processes that queue. There is, of course, always a risk that you might close a bug when you're in a tunnel or on a plane without WiFi and when you later get back online the sync fails because of some conflict.

The reason I built this was partly to scratch an itch I had ("What's the ideal way possible for me to use Bugzilla?") and also to experiment with some new techniques, namely AngularJS and localforage.

So, the way it works is:

1. You pick your favorite product and components.

2. All bugs under these products and components are downloaded and stored locally in your browser (thank you localforage).

3. When you click any bug it then proceeds to download its change history and its comments.

4. Periodically it checks each of your chosen product and components to see if new bugs or new comments have been added.

5. If you refresh your browser, all bugs are loaded from a local copy stored in your browser and in the background it downloads any new bugs or comments or changes.

6. If you enter your username and password, an auth token is stored in your browser and you can thus access secure bugs.

Pros and cons

The main advantage of Buggy compared to Bugzilla is that it's fast to navigate. You can instantly filter bugs by status(es), components and/or by searching in the bug summary.

The disadvantage of Buggy is that you can't see all fields, file new bugs or change all fields.

The code

The code is of course open source. It's available on https://github.com/peterbe/buggy and released under a MPL 2 license.

The code requires no server. It's just an HTML page with some CSS and Javascript.

Everything is done using AngularJS. It's only my second AngularJS project but this is also part of why I built this. To learn AngularJS better.

Much of the inspiration came from the CSS framework Pure and one of their sample layouts which I started with and hacked into shape.

The deployment

Because Buggy doesn't require a server, this is the very first time I've been able to deploy something entirely on CDN. Not just the images, CSS and Javascript but the main HTML page as well. Before I explain how I did that, let me explain about the make.py script.

I really wanted to use Grunt but it just didn't work for me. There are many positive things about Grunt such as the ease with which you can easily add plugins and I like how you just have one "standard" file that defines how a bunch of meta tasks should be done. However, I just couldn't get the concatenation and minification and stuff to work together. Individually each tool works fine, such as the grunt-contrib-uglify plugin but together none of them appeared to want to work. Perhaps I just required too much.

In the end I wrote a script in python that does exactly what I want for deployment. Its features are:

• Hashes in the minified and concatenated CSS and Javascript files (e.g. vendor-8254f6b.min.js)
• Custom names for the minified and concatenated CSS and Javascript files so I can easily set far-future cache headers (e.g. /_cache/vendor-8254f6b.min.js)
• Ability to fold all CSS minified into the HTML (since there's only one page, theres little reason to make the CSS external)
• A Git revision SHA into the HTML of the generated ./dist/index.html file
• All files in ./client/static/ copied intelligently into ./dist/static/
• Images in CSS to be given hashes so they too can have far-future cache headers

So, the way I have it set up is that, on my server, I have a it run python make.py and that generates a complete site in a ./dist/ directory. I then point Nginx to that directory and run it under http://buggy-origin.peterbe.com. Then I set up a Amazon Cloudfront distribution to that domain and then lastly I set up a CNAME for buggy.peterbe.com to point to the Cloudfront distribution.

The future

I try my best to maintain a TODO file inside the repo. That's where I write down things to come. (it's also works as a changelog) since I also use this file to write down what's been done.

One of the main features I want to add is the ability to add bugs that are outside your chosen products and components. It'll be a "fake" component called "Misc". This is for bugs outside the products and components you usually monitor and work in but perhaps bugs you've filed or been assigned to. Or just other bugs you're interested in in general.

Another major feature to work on is the ability to choose to see more fields and ability to edit these too. This will require some configuration on the individual users' behalf. For example, some people use the "Target Milestone" a lot. Some use the "Importance" a lot. So, some generic solution is needed to accomodate all these non-basic fields.

And last but not least, the Bugzilla team here at Mozilla is working on a very exciting project that allows you to register a certain list of bugs with a WebSocket and have it push to you as soon as these bugs change. That means that I won't have to periodically query bugzilla every 30 seconds if certain bugs have changed but instead get instant notifications when they do. That's going to be major! I confidently speculate that that will be implemented some time summer this year.

Give it a go. What are you waiting for? :) Go to http://buggy.peterbe.com/, pick your favorite products and components and try to use it for a week.

I do not deny it. I'm a YouTube fiend. I very rarely watch YouTube on my computer but a lot on my Apple TV and only tablet. It's

Here are some of my favorite YouTube channels that I subscribe to and encourage you to do the same if you aren't already and if there's something it appears you'll like too.

1. MinutePhysics

They started as clips that were around 1 minute but are now of variable length. I just adore Henry's voice and the topics he chooses. The animations are cute and even though seasoned with silly cat and dog references they really help to explain some of the most advanced subjects in physics.

Incidentally, this was the first channel I subscribed to once I figured that's the best way to get recurring content from channels I really liked.

2. Numberphile

This is a Brady Haran production that speaks directly to my mathematical aspirations. These aspirations aren't to solve any complex calculus problems but to keep that almost mystic infatuation alive I have with mathematics. There's something wonderfully down to earth and kind about the content which challenge you without patronizing you. By the way, my favorite interviewee, James Grime has his own channel now called singingbanana and also, by the way, and amazingly unattractive website.

3. Veritasium

Derek Muller is a brilliant video maker. Most of his videos are about science and it's mainly Derek holding his camera at arms length filming his pleasant face and talking about the perception or understanding of science. More so than the science itself. Actually some videos are not about how people (miss)understand science but speak directly to you and those are just brilliant. Usually sufficiently advanced to really get reallying thinking hard.

4. CGP Grey

The only, of my top favorite channels, that is not about natural science. These videos are on social science subjects you might never have thought to think about and not only that, but each and every one digs deep and misses very few facts. Similarly to SciShow, these videos require your full attention. Because what you learn from them is often so very valuable, I've revisited many videos. Some more than twice.

5. MinuteEarth

This is Henry Reich's (see above about MinutePhysics) second channel and the name of the channels fully describes what the videos are about. The animations are really magnificantly simple and rich at the same time. The subject matters in this videos are generally less advanced that those in MinutePhysics but often full of really interesting factoids to keep up your sleeve for dinner parties.

6. SciShow

Hang Green is a gem! His geeky and passionate mannerisms is worth it just on its own. But you have to pay full attention because Hank speaks very fast. There is though an important undertone that isn't immediately obvious. There is this feeling of deeply researched facts. Even though you only understand a small part of it all (not to mention how little you remember!) it's inspiring that someone takes the time to do all the research.
A lot of subject matters are science oriented but more popular sciencey.

7. Sixty Symbols

Another Brady Haran production, but this time more about physics and than Numberphile which is more about mathematics. Almost all videos are Brady interviewing doctors and professors in physics at the University of Nottingham. All very humble and approachable interviewees that, perhaps thanks to Brady's brilliant questions, the subjects are understandable but also very exciting because they're usually on matters that are very advanced and something more to look forward to than to enjoy in the moment.

8. Piled Higher and Deeper (PHD Comics)

This is a newcomer and I include it because they're of such high quality and adorable animations. To be honest I don't think I really understand what the various videos have in common. For example, one recent video is on quantum entanglement and another on the Dead Sea scrolls. Either way, every video is professional and highly enjoyable.

There are more channels I subscribe to and enjoy very much but the above list are my favorite ones. For example, I watch Jamie Oliver's Food Tube videos just as often but that's somehow more "obvious".

Actually I have many more channels on science and a bunch of computers and programming but I'm just simply not as passionate about them as I are with the channels mentioned above.

I really hope that by writing this it will inspire one or two fellow science nerdy readers to also discover some of the channels mentioned here.