Butter smooth timeline for your video/audio project (canvas drawing)

Conceived as a pet project for editing TV schedule for small IPTV providers, the project proved to be an interesting challenge.

For starter we looked at some other implementations and liked this one: it comes with lots of examples and proved to be interesting enough to work with. However we quickly realized that the code does not scale well at all. All the examples feature very short videos and we needed a visual time representation for at least 24 hours period. Basically the code would have drawn each second on the canvas no matter how long the period is and thus was working progressively slower with larger periods of time being used. It also binds directly to a slider / scroller and thus was updating on value change, instead of when the browser has a drawing frame schedule.

Never the less the code was a good example of a timeline view executed with canvas and we decided to use its concepts.

The first thing to do was to make it closure compiler friendly, this was easy. We also stripped the binding with the scales / scrollers and instead exposed methods to set the values on the class regardless of the means used for the user interface.

Then we started looking for optimization opportunities. We notices that if we draw all the seconds in the period on the canvas we quickly turn the time lines (scale separators) into a blank line - this is because all the drawing is always performed: each second's position is determined and drawn on the canvas. As you can imagine for 24 hours period those draws were often from thousands to tens of thousands. We decided it would be more wise to separate the whole period into dividable frames and use those as steps when jumping from one time position to the next instead of iterating on every second of the period.

Our first solution was fine, but iterating over the same amount of time using different steps has two side effects: number one is related to how canvas use the dimensions internally. Basically for best results you would want to set the canvas size to the actual size of the view. Also when drawing single pixel wide lines you would want to shift the line to a half a pixel (i.e if you want to draw on pixel one you need to perform a stroke at position 0.5 and 1 pixel wide). If you draw on anything else anti-aliasing kicks in and what you see is not as crisp as you might hoped for. Because we implemented this half pixel shifting as the first thing in our code the defect introduced with the multiple walk trough the scale was not visible: basically we were drawing on the same pixel multiple times. Still - the code was much faster now, because at this point we did not draw on every second but instead on every time the next time point would stand at least few pixels away from the previous. Considering the capabilities of the modern monitors this would mean one would draw not more than few thousand lines, more often than never less than one thousand, regardless of the time frame to be visualized. The second problem was to manage the time stamps on the top - managing meaningful time stamps required much more code.

Using the shifting to a whole pixel drawing technique, we also need to consider this shifting when drawing the separate program items and that proved to be not so trivial, because while using steps in drawing the scale, using those when drawing precise time is not perfect. Instead of putting too much effort in aligning the pixels, we instead decided to investigate another alternative: let the canvas use fractions when drawing but keep the reduced drawing calls by using time steps larger than 1.

This however allowed us to see the defect introduced on the previous step: going on top of a fraction of pixels does make a difference in the visualization  and because we draw the different steps sizes in different length the resulting image was very poor. Basically the drawing is performed on the same fraction (i.e. 0.232536) but visually you get a wider line because you have drawn twice on it.

At this point we were not happy with neither pixel alignment results (too difficult to shift everything to a full number, too much calculations) nor step cycling results (poor picture quality).

During the weekend I had the idea of removing the cycles of drawing with different step and instead use only one step but instead calculate the step offset to the predefined steps. If modulus division matches 0 then we are on a certain step (i.e. full hour if we divide to 3600). Also one thing to notice was that we do not actually need to calculate the X position to draw on every step: because the step is determined only once per redraw we can also determine the difference between two adjacent steps use it increment the X position without actually calculating X for a certain second in time. As a final optimization I suggested also to calculate the first visible second and align it to the step (so we skip cycling over x positions that are out of view) So basically you calculate the X for the 0 second in the period, then for the first step value and using the offset to 0 (the start of the visible area) you have the initial visible second and from it you calculate the first draw-able X value. From then on you only increment the second to draw by the time step and the x value for it by the difference between two values.

Now the code behaved perfectly, even on Atom powered laptops with Chrome browser. However in Firefox things were not that brilliant. There was noticeable lag on updates on desktop Core processor, let alone a netbook.

Next optimization we had to perform was to decouple the drawing of the canvas from the scale and scroll updates. This is easily done and well documented technique (just look up "using request animation frame" in google). Basically every time we have value updates we set a dirty flag and call the RAF. It calls itself until the dirty flag is turned off.

Now that was a butter smooth canvas visualization.

What was accomplished: unlimited time frame to visualize, unlimited items in the schedule, decoupled value updates (i.e. you can set them on batches, it will not hurt performance), smooth animation and great performance on all modern browsers.

What we learned: drawing on the canvas is expensive, draw as little as possible, calculations with floating number are also expensive, reduce those to the possible minimum, use RAF to limit the drawing operation to the browser optimized frame rate.

A demo application will be available soon, as well as the code related to the time line will be on GitHub once the project is complete.