Using HTML5 MediaSource API as Video Source

More than half a year ago I developed a HTML5 Video Player that is available on github. The webserver (in our case lighttpd) accepts (byte) range requests. So we can feed the source (mp4, webm or ogg) directly as a source to the player/video element. So the video gets loaded as needed. This worked great for the last months and works automatically.

I recently came across a problem with Internet Explorer 11 on Windows 8.1. The browser does not play the video. It completely freezes instantly or after a few seconds. There is no chance to view the video except if you disable byte range request and load the video completely. It does this even on the popular videoplayer websites like flowplayer and jwplayer. But it didnt happen with YouTube videos.
After doing some research I found out that the source of the video is not a single URL but a blob (blob:<identifier>). youtube.com/html5 shows the availability of HTML5 Video in your Browser and it also shows the availability of a feature called Media Source Extension (MSE). The MSE is used as a source for the HTML Media Elements (like <video> and <audio>).

Note that MSE is not available in IE11 on Windows 7 (with identical IE version as on Windows 8.1) for unknown reason.

The procedure for using MSE is the following:

  1. Create a new MediaSource (MS) object
  2. Create a blob URL for the MS object
  3. Set src of <video> to the blob URL
  4. Wait for the ‘sourceopen’ event to be fired on MS
  5. Create a new SourceBuffer (SB)
  6. Append data to the SB periodically

This works great for webm and ogg files. The only problem here is how do determine when new data has to be loaded.

Mp4 video files cannot be used in general for the MSE purposes. The keyword here is MPEG DASH (Dynamic Adaptive Streaming over HTTP). This is a technique that allows two things.

  1. It allows to split a mp4 file into segments that can be feed into the MS object
  2. It allows to switch quality while playing without the need two stop playback.

1) Is what we are most interested in because we want to load mp4 video data periodically if needed.

2) is a nice side effect that you may have encountered as well. When switching quality of a video on YouTube, you might have noticed that it take a few seconds until the video has better quality. This is because the current fragment will be played from the SB and after that the fragment with better quality is loaded and played.

I used GPAC’s MP4Box to create a fragmented mp4 file, that can be used for streaming. Mp4Box also created a XML file that contains the fragments. Not only the byte ranges are included but also the time offset in the video files. This way you can load a new segment when the segment is almost finished playing (e.g. 80% of the time of the segment is reached).

I hope this gives you a short introduction. I will post more blog posts when I have some demos/code I can present.

Mixing styles from an attribute and .style in dart in a polymer element

I came across this behaviour by chance. Imagine the following code that is used in a custom polymer element:

...
<polymer-element...>
<template>
...
<div id="foo" style="width: {{ totalWidth }}px;"></div>
...
</template>
</polymer-element>
...

And in Dart I tried the following:

$['foo'].style.backgroundImage = 'url('+imgAsBase64+')';

I tried all different syntaxes of URL()… but never got it working. What happens here is pretty simple: I tried to set the backgroudn image via the style attribute before the style attribute was evaluated by polymer. So the re-rendering (because I changed totalWidth also) overwrites my changes. Mixing is complicated that’s why I finally put it all into the HTML Code:

...
<polymer-element...>
<template>
...
<div id="foo" style="width: {{ totalWidth }}px; background-image: url({{ img }});"></div>
...
</template>
</polymer-element>
...

Spawning isolates in Dart

Spawning new isolates in Dart is easy if you know how to do it:

main.dart

import 'dart:isolate';
import 'dart:async';

void main() {
  Future isolate1 = Isolate.spawnUri(Uri.parse('isolate.dart'), [], null);
  Future isolate2 = Isolate.spawnUri(Uri.parse('isolate.dart'), [], null);
  Future.wait([isolate1,isolate2]);
  while(true);
}

Future.wait only waits until the two isolates are spawned. If the main application stops running, the two spawned isolates will stop running as well. That’s why we need a while(true)-loop here.

The isolate itself looks like this:

library isolate;

void main() {
  print('isolate.main()');
  while(true);
}

The two arguments after the URI are used for a list of arguments and an optional message. If you want to use this, just modify the main method of your isolate:

void main(args) {...}
void main(args, message) {...}

MySQL: ORDER BY RAND() – a case study of alternatives

I finally had the time to translate my post from german to english – so here we go :)

Das Problem

Many of you reading this post might already stopped by this problem – you just want a few random rows from your database and most of the times you might have used “ORDER BY RAND()” and not thought about, what’s happening there. Just add all the columns you want to select, add JOINS, WHERE, etc. and add ORDER BY RAND() and LIMIT.

For small applications this might even scale well, but what happens in large applications with hundreds, thousands or millions of rows in a table?

I want to look a little closer into this. While developing my deprecated rhCMS (a content management system), I asked myself the same question.

Weiterlesen

[Dart+Polymer] Styling Elements

Styling custom elements is sometimes tricky. In general the website styles do NOT apply to the custom element. There is a special attribute called “apply-author-styles” that can be set to “true” to apply your website’s styles to your custom element.

In general each custom element should have its  styles separated from the website so it can be reused easily. Let’s have a look at a simple example:

Weiterlesen

Animatr – GUI & Structure: Nesting Polymer Elements

The first step of rebuilding animatr was to remove all code and totally rebuild the GUI using a single custom element: <animatr-gui> using polymer. The GUI itself has three nested custom elements:

<animatr-titlebar> – Containins the windows’ title, close button, …
<animatr-toolbars> – Wrapper for the toolbars
<animatr-panels> – A wrapper to create a GUI using pannels that can be dragged.

The HTML code of the root element looks like this:

<head>
  <link href="packages/animatr_gui/elements/animatr-ui.html" rel="import" />
</head>
<body>
  <animatr-ui></animatr-ui>
</body>

The corresponding element HTML code is:

<polymer-element name="animatr-ui">
  <meta charset="utf-8"/>
  <link rel="import" href="animatr-titlebar.html">
  <link rel="import" href="animatr-toolbars.html">
  <link rel="import" href="animatr-panels.html">
  <link rel="import" href="animatr-panel-....html">
  <template>
      <animatr-titlebar id="animatr-titlebar"></animatr-titlebar>
      <animatr-toolbars id="animatr-toolbars"></animatr-toolbars>
      <animatr-panels id="animatr-panels"></animatr-panels>
  </template>
  <script type="application/dart" src="animatr-ui.dart"></script>
</polymer-element>
 

The Dart code for the custom element is:


import 'package:polymer/polymer.dart';

import 'dart:html';

import 'package:animatr_gui/elements/animatr-titlebar.dart';
import 'package:animatr_gui/elements/animatr-toolbars.dart';
import 'package:animatr_gui/elements/animatr-panels.dart';

import 'animatr-panel-....dart';

@CustomTag('animatr-ui')
class AnimatrUiElement extends PolymerElement {
  bool get applyAuthorStyles =&gt; true;

  ...PanelElement panel_...;

  AnimatrUiElement.created() : super.created() {
    panel_... = document.createElement('animatr-panel-...');
  }

  AnimatrTitlebarElement titlebar;

  AnimatrToolbarsElement toolbars;

  AnimatrPanelsElement panels;

  @observable Project currentProject;

  @override
  void enteredView() {
    super.enteredView();

    titlebar = $['animatr-titlebar'];
    toolbars = $['animatr-toolbars'];
    panels = $['animatr-panels'];

    titlebar.ui = this;
    panels.ui = this;

    panels.addPanel(panel_..., 'north/east/south/west/center');
   }
}

Nesting elements was complicated first. Assuming you have your project’s root file (mine is called animatr.html), you could import all elements in this file, at a central place. But this had some strange behaviour to me:
The outer elements get created first and thus, before all nested elements. Thus I wasn’t able to correctly query for the nested elements. Because the custom element wasn’t registered Google Dart had some type errors.

The solution was to import other elements within the custom element itself, right before the <template> tag.

I had another problem using special characters. It turned out that the charset settings (e.g. by using <meta charset=”utf-8″/>) is not propagated to custom elements. Thus make sure to include a <meta>-tag in your custom elements if you want to use custom elements.

Animatr – A first introduction

In this post I want to briefly introduce you to my most recent project called “animatr”. Animatr is a tool to create canvas animations. I will focus on advertisements but in theory you can create all types of animations and sizes. Even playing a video within the animation is possible.

The first version was developed about a year ago and was based on fabric.js But very soon I felt the framework limited me to their features. So I quickly decided to rewrite it as a web app on my own. Due to some specific webkit/chrome features I decided to finally rewrite the project using Google’s Dart and to make an app out of it using node-webkit.

The latest and third version is based on Google Dart, Node-Webkit and Polymer. While polymer is the replacement for web-ui.

I will write follow up blogs about concepts and code I use while developing animatr and give you examples. So far I can tell that detailed examples are rare. That’s why I felt to blog about it myself.

My blog

I have been thinking for years now if I should start blogging or not. Finally I decided to do so :)

I will be blogging about all web topics I will deal with. Nevertheless I want to focus on Dart + Polymer I have been working on recently.