Load Protocol Buffers in the Browser

While working on a project recently I was inspired to learn a bit more about Protobuf files. I especially wanted to know how you can load them directly in a browser.

Protocol Buffers, also called protobuf for short, are a method of serializing structured data. Generally they are not used for transferring data to a browser because for large datasets the browser may take longer to decode them than standard JSON, but they are definitely used in the browser for some things like mapping vector tiles.

The Decoding Process

In our case, the data being sent as part of the feed was not super large, so decoding it in the browser shouldn’t be too taxing. Let’s try it!

Step 1 is only done once (unless your schema changes): convert the schema, typically a .proto file, into a JavaScript module. The stream I was working with was a GTFS Realtime Stream, and I found the .proto file here. To convert this to a JavaScript module that I could use in a browser, I used the pbf package. Install it globally then use it to translate:

npm install -g pbf
pbf gtfs-realtime.proto --browser > gtfs-realtime.browser.proto.js

It will assume a Node.js running environment unless you specify --browser.

Next, build your HTML/JavaScript page. Include the pbf library and the file you generated above (gtfs-realtime.browser.proto.js) on the page. You can use Fetch or any other request method to get your data. A few notes here though:

  1. If you’re using a 3rd-party API like I was, your feed is most likely not on your same domain. You’ll have to ensure there is a proper Access-Control-Allow-Origin header on the feed so you can fetch it properly.
  2. Your browser might cache the feed - you may need to add a cache-busting URL parameter to avoid this.
  3. Ensure you’re processing the data as an ArrayBuffer. For Fetch this means using response.arrayBuffer(); (or use xhr.responseType = "arraybuffer"; if using XMLHttpRequest directly).

After you have the data, you then call a function from your generated .proto.js file. This will be different depending on your Protocol Buffer schema. In my example I’m using the FeedMessage object, so:

const obj = FeedMessage.read(pbf);
return obj.entity;

This will be the JavaScript object that you can now use in your code.

Next Step: Map It

For the example feed I was working with, it contained live bus locations, so it would be great to place these locations on a map. This process was pretty straightforward, because the JSON that we were returning above has a latitude and longitude for each entry. The only magic I had to do was handle the auto-updating. Instead of doing something elegant like updating the locations of existing points, I just created a client-side FeatureLayer and used setInterval to clear out all the data, re-query the feed, and re-create all the points on a regular basis:

setInterval(() => {
  updateLayer(fl, layerView);
}, 3000);

In this case updating the map every three seconds (much more often than the feed provides data updates), so sped-up, looks like this:

… and here is the code:

How are you using Protobufs? Let me know.

Written by Gavin Rehkemper on 
Last updated on 

Get an email summary of my blog posts (four per year):

Join email newsletter

... or follow the blog here:

See Also