"Today I Saw the Future",
of video

A presentation by @darcy

TBD - Too Bad Darcy!!!!!!!!!!!!!!!!!!!

A presentation by @darcy

Thanks to the organizers,
sponsors & jQuery Team!

Darcy Clarke

Designer, Developer, Speaker, Mentor, Consultant

Tweet @darcy

Fork @darcyclarke

Visit darcyclarke.me

Work

  • Co-Founded Themify
  • Fi (Fantasy Interactive), Jet Cooper, Polar Mobile, Say Yeah!, Playground, Blast Radius, Bnotions, Daily Challenge, Pinpoint Social, My City Lives, Viafoura, Flixel
  • Google, Microsoft, Nike, Samsung, Porsche, Ducati, Disney, Red Bull, CBS, Panasonic
  • jQuery, DSS, Frontend Developer Interview Questions, Watch.js, Repo.js ...

jQueryTO 2013

Documenting
User Interfaces & Interactions

Toronto

London

Prague

San Francisco

My Last 12 Months

  • Documentation
  • Open Source
  • Standards & Accessibility
  • Progressive Enhancement
  • Feature Detection & Polyfills
  • Build Tools
  • Server-side Rendering
  • Language Abstractions & Preprocessors
  • Typography
  • Responsive Images
  • * Video

We'll Cover

History, Case Study, Libraries, The Future

History

Codecs, Containers & Playback

1986 - Sony release D1

  • First digital video recording format
  • Stored 94 minutes
  • Uncompressed

1988 - Sony & Ampex release D2

  • Offered simultaneous playback
  • Stored composite video over component
  • Still uncompressed

1988 - H.261 is released

  • First member of the H.26x family
  • Most all subsequent video coding standards are based on it
  • Used compression

Fun Fact: "Codec"

"Compression / Decompression"

Let's replay 1991 - 2007

1991

1995

1996

1997

The Aftermath...

Fragmentation

February 28th, 2007

Opera Proposes <video> element

~ lists.whatwg.org/pipermail/whatwg-whatwg.org/2007-February/009702.html

Implements it that same day

7 years later

Implementation

<video controls>
  <source src="video.mp4" type="video/mp4;">
  <source src="video.webm" type="video/webm;">
  <source src="video.ogv" type="video/ogg; codecs=theora, vorbis">
  <a href="video.mp4">Download Video</a>
</video>

2014

Requirements

  • Easy to create at scale
  • Small file size
  • Highest quality possible
  • Works on all desktop browsers aswell as mobile devices
  • Video must autoplay & loop
  • Sound not required

Also,
Kill the GIF!

Initial Thoughts

  • Focus on WEBM
  • Investigate JavaScript-based solutions utilizing canvas for mobile

Research

JavaScript Decoders

H264 Broadway.js

  • github.com/mbebenita/Broadway
  • Demos didn't work
  • Code didn't work
  • Narrow file requirements
  • Required significant changes to asset pipeline

MPEG1: JSMPEG

  • github.com/phoboslab/jsmpeg
var canvas = document.getElementById('canvas');
var player = new jsmpeg('video.mpg', {canvas:canvas, autoplay:true, loop: true });

MPEG1: JSMPEG

  • github.com/phoboslab/jsmpeg
  • Narrow file requirments
  • Picky implementation
  • MPEG1 is a poor/old format in general

APNG: PNG.js

  • github.com/devongovett/png.js
var canvas = document.getElementById('canvas');
PNG.load('video.png', canvas);

APNG: PNG.js

  • github.com/devongovett/png.js
  • APNG is bloated
  • Tools to generate files were painful
  • Required significant changes to asset pipeline

WEBP: libwebpjs.js

  • http://libwebpjs.appspot.com/v0.1.3
  • Animated WEBP is still in it's infancy
  • No tools to generate the files

WEBM: vp8-webm-js-decoder

  • libwebpjs.hohenlimburg.org/vp8/webm-javascript-decoder-2
  • github.com/darcyclarke/vp8-webm-javascript-decoder
var canvas = $('canvas')[0];
var jqXHR = $.ajax({
  type: 'GET',
  mimeType: 'text/plain; charset=x-user-defined',
  url: 'video.webm'
});
jqXHR.then(function(data){
  data = data.split('').map(function(e){
    return e.charCodeAt(0) & 0xff;
  });
  var player = new webm(canvas);
  player.run(data);
});

WEBM: vp8-webm-js-decoder

  • libwebpjs.hohenlimburg.org/vp8/webm-javascript-decoder-2
  • Poorly written
  • Performance on desktop ~20-30fps
  • Performance on mobile <10fps

Sprite PNG + JSON: anim_encoder

  • github.com/sublimehq/anim_encoder

Conclusions

  • Processing overhead is worth the gain if it means smaller file size or higher quality
  • All motion graphics formats are sh*t
  • All file decoders are experimental, poorly documented, broken, polyfills
  • But also really cool
  • Mobile performance vetos all
  • The simpler the better

Solution: HTML5 Video Player that fallsback to GIF

Problem: Mobile devices break the video implementation

Determine Support

Feature Detection - Codecs

modernizr.com

if( Modernizr.video ) {
  // browser supports video
}
Modernizr.video.h264 // 'probably', 'maybe', ''
Modernizr.video.webm
Modernizr.video.vp9
Modernizr.video.ogg

Feature Detection - Codecs

var video = (function() {
  var elem = document.createElement('video');
  var bool = false;
  try {
    if ( bool = !!elem.canPlayType ) {
      bool = new Boolean(bool);
      bool.ogg = elem.canPlayType('video/ogg; codecs="theora"').replace(/^no$/,'');
      bool.h264 = elem.canPlayType('video/mp4; codecs="avc1.42E01E"').replace(/^no$/,'');
      bool.webm = elem.canPlayType('video/webm; codecs="vp8, vorbis"').replace(/^no$/,'');
      bool.vp9 = elem.canPlayType('video/webm; codecs="vp9"').replace(/^no$/,'');
    }
  } catch(e){}
  return bool;
})();

Feature Detection - Loop

var elem = document.createElement('video');
if( 'loop' in elem ) {
  // browser supports looping
}

Feature Detection - Autoplay

var video = (function() { ... })();
var autoplay;
var timeout;
var waitTime = 300;
var elem = document.createElement('video');
var elemStyle = elem.style;
var testAutoplay = function(called) {
  clearTimeout(timeout);
  elem.removeEventListener('playing', testAutoplay);
  autoplay = called || elem.currentTime !== 0;
  elem.parentNode.removeChild(elem);
};
 
if (!('autoplay' in elem)) {
  autoplay = false;
  return;
}
 
elemStyle.height = 0;
elemStyle.width = 0;
 
try {
  if (video.h264) {
    elem.src = 'data:video/mp4;base64,...';
  }
  else if (video.ogg) {
    elem.src = 'data:video/ogg;base64,...';
  }
  else {
    autoplay = false;
    return;
  }
}
 
catch (e) {
  autoplay = false;
  return;
}

Duck Typing

window.isOld = document.getElementsByTagName('html')[0].className.indexOf('lte-ie9') > -1;
window.isMobile = !!(function(a){return !!(/(android|bb\d+|meego).+mobile|avantgo|bada\/|blackberry|blazer|compal|elaine|fennec|hiptop|iemobile|ip(hone|od)|iris|kindle|lge |maemo|midp|mmp|mobile.+firefox|netfront|opera m(ob|in)i|palm( os)?|phone|p(ixi|re)\/|plucker|pocket|psp|series(4|6)0|symbian|treo|up\.(browser|link)|vodafone|wap|windows (ce|phone)|xda|xiino/i.test(a)||/1207|6310|6590|3gso|4thp|50[1-6]i|770s|802s|a wa|abac|ac(er|oo|s\-)|ai(ko|rn)|al(av|ca|co)|amoi|an(ex|ny|yw)|aptu|ar(ch|go)|as(te|us)|attw|au(di|\-m|r |s )|avan|be(ck|ll|nq)|bi(lb|rd)|bl(ac|az)|br(e|v)w|bumb|bw\-(n|u)|c55\/|capi|ccwa|cdm\-|cell|chtm|cldc|cmd\-|co(mp|nd)|craw|da(it|ll|ng)|dbte|dc\-s|devi|dica|dmob|do(c|p)o|ds(12|\-d)|el(49|ai)|em(l2|ul)|er(ic|k0)|esl8|ez([4-7]0|os|wa|ze)|fetc|fly(\-|_)|g1 u|g560|gene|gf\-5|g\-mo|go(\.w|od)|gr(ad|un)|haie|hcit|hd\-(m|p|t)|hei\-|hi(pt|ta)|hp( i|ip)|hs\-c|ht(c(\-| |_|a|g|p|s|t)|tp)|hu(aw|tc)|i\-(20|go|ma)|i230|iac( |\-|\/)|ibro|idea|ig01|ikom|im1k|inno|ipaq|iris|ja(t|v)a|jbro|jemu|jigs|kddi|keji|kgt( |\/)|klon|kpt |kwc\-|kyo(c|k)|le(no|xi)|lg( g|\/(k|l|u)|50|54|\-[a-w])|libw|lynx|m1\-w|m3ga|m50\/|ma(te|ui|xo)|mc(01|21|ca)|m\-cr|me(rc|ri)|mi(o8|oa|ts)|mmef|mo(01|02|bi|de|do|t(\-| |o|v)|zz)|mt(50|p1|v )|mwbp|mywa|n10[0-2]|n20[2-3]|n30(0|2)|n50(0|2|5)|n7(0(0|1)|10)|ne((c|m)\-|on|tf|wf|wg|wt)|nok(6|i)|nzph|o2im|op(ti|wv)|oran|owg1|p800|pan(a|d|t)|pdxg|pg(13|\-([1-8]|c))|phil|pire|pl(ay|uc)|pn\-2|po(ck|rt|se)|prox|psio|pt\-g|qa\-a|qc(07|12|21|32|60|\-[2-7]|i\-)|qtek|r380|r600|raks|rim9|ro(ve|zo)|s55\/|sa(ge|ma|mm|ms|ny|va)|sc(01|h\-|oo|p\-)|sdk\/|se(c(\-|0|1)|47|mc|nd|ri)|sgh\-|shar|sie(\-|m)|sk\-0|sl(45|id)|sm(al|ar|b3|it|t5)|so(ft|ny)|sp(01|h\-|v\-|v )|sy(01|mb)|t2(18|50)|t6(00|10|18)|ta(gt|lk)|tcl\-|tdg\-|tel(i|m)|tim\-|t\-mo|to(pl|sh)|ts(70|m\-|m3|m5)|tx\-9|up(\.b|g1|si)|utst|v400|v750|veri|vi(rg|te)|vk(40|5[0-3]|\-v)|vm40|voda|vulc|vx(52|53|60|61|70|80|81|83|85|98)|w3c(\-| )|webc|whit|wi(g |nc|nw)|wmlb|wonu|x700|yas\-|your|zeto|zte\-/i.test(a.substr(0,4)))})(navigator.userAgent||navigator.vendor||window.opera);

Final Result

<iframe src='//flixel.com/player/0h5hym95isieesl7cxg6' frameborder='0' allowfullscreen></iframe>

Today I Saw the Future

May 3rd, 2013

Brendan Eich

Mozilla and OTOY deliver the power of native PC applications to the Web, unveil next generation JavaScript video codec for movies and cloud gaming

~ brendaneich.com/2013/05/today-i-saw-the-future

What makes this interesting?

ORBX.js Features

  • 25% better compression than H.264
  • Adaptive bit-rate (while streaming)
  • Better color depth
  • Better intra-frame coding
  • IP-blind runtime
  • Flexibile
  • Open Source

The GPU cloud has your back. Think of the amazing 3D games that we have on PCs, consoles, and handheld devices thanks to the GPU. Now think of hundreds of GPUs in the cloud, working for you...

~ brendaneich.com/2013/05/today-i-saw-the-future

Questions

  • JavaScript
  • Canvas
  • WebGL
  • Cloud
  • Streaming
  • Sockets
  • Performance
  • Support
  • Flash

Where's the repo?

This sounds familiar...

Famo.us is a free and open source JavaScript development framework backed by a host of cloud services. Famo.us is currently in private beta and about to launch into open beta to over 60,000 developers.

~ angel.co/famo-us/jobs

Update: December 18th, 2013

We launched the ORBX and OctaneCloud AMIs on the AWS Marketplace

~ render.otoy.com/newsblog/?p=317

But there's more

  • ORBX.js will be free forever
  • Will be Open Sourced (potentially as early as this summer)
  • Major studios are looking at ORBX to solve video-on-demand watermarking (DRM free)

Not right now...

So What is the Future of Video?

  • MP4
  • Open Source Format like WEBM
  • JavaScript (Brendan Eich swears by it)
  • Cloud
  • ORBX.js
  • * Digital Rights Management
  • * Encrypted Media Extensions