It's been a busy few several months, but now that we have some breathing
room, I wanted to take stock of what we have done over the last year or so.
This is a good thing for most people and companies to do of course, but being a scrappy, (questionably) young organisation, it's doubly important for us to introspect. This allows us to both recognise our achievements and ensure that we are accomplishing what we have set out to do.
One thing that is clear to me is that we have been lagging in writing about some of the interesting things that we have had the opportunity to work on, so you can expect to see some more posts expanding on what you find below, as well as some of the newer work that we have begun.
(note: I write about our open source contributions below, but needless to say, none of it is possible without the collaboration, input, and reviews of members of the community)
WHIP/WHEP client and server for GStreamer
If you're in the WebRTC world, you likely have not missed the excitement around standardisation of HTTP-based signalling protocols, culminating in the WHIP and WHEP specifications.
Tarun has been driving our client and server
implementations for both these protocols, and in the process has been
refactoring some of the webrtcsink and webrtcsrc code to make it easier to
add more signaller implementations. You can find out more about this work in
his talk at GstConf 2023
and we'll be writing more about the ongoing effort here as well.
Low-latency embedded audio with PipeWire
Some of our work involves implementing a framework for very low-latency audio processing on an embedded device. PipeWire is a good fit for this sort of application, but we have had to implement a couple of features to make it work.
It turns out that doing timer-based scheduling can be more CPU intensive than ALSA period interrupts at low latencies, so we implemented an IRQ-based scheduling mode for PipeWire. This is now used by default when a pro-audio profile is selected for an ALSA device.
In addition to this, we also implemented rate adaptation for USB gadget devices using the USB Audio Class "feedback control" mechanism. This allows USB gadget devices to adapt their playback/capture rates to the graph's rate without having to perform resampling on the device, saving valuable CPU and latency.
There is likely still some room to optimise things, so expect to more hear on this front soon.
Compress offload in PipeWire
Sanchayan has written about the work we did to add support in PipeWire for offloading compressed audio. This is something we explored in PulseAudio (there's even an implementation out there), but it's a testament to the PipeWire design that we were able to get this done without any protocol changes.
This should be useful in various embedded devices that have both the hardware and firmware to make use of this power-saving feature.
GStreamer LC3 encoder and decoder
Tarun wrote a GStreamer plugin implementing the LC3 codec
using the liblc3 library. This is the
primary codec for next-generation wireless audio devices implementing the
Bluetooth LE Audio specification. The plugin is upstream and can be used to
encode and decode LC3 data already, but will likely be more useful when the
existing Bluetooth plugins to talk to Bluetooth devices get LE audio support.
QUIC plugins for GStreamer
Sanchayan implemented a QUIC source and sink plugin in Rust, allowing us to start experimenting with the next generation of network transports. For the curious, the plugins sit on top of the Quinn implementation of the QUIC protocol.
There is a merge request open that should land soon, and we're already seeing folks using these plugins.
AWS S3 plugins
We've been fleshing out the AWS S3 plugins over the years, and we've added a
new awss3putobjectsink. This provides a better way to push small or sparse
data to S3 (subtitles, for example), without potentially losing data in
case of a pipeline crash.
We'll also be expecting this to look a little more like multifilesink,
allowing us to arbitrary split up data and write to S3 directly as multiple
objects.
Update to webrtc-audio-processing
We also updated the webrtc-audio-processing
library, based on more recent upstream libwebrtc. This is one of those things
that becomes surprisingly hard as you get into it -- packaging an API-unstable
library correctly, while supporting a plethora of operating system and
architecture combinations.
Clients
We can't always speak publicly of the work we are doing with our clients, but there have been a few interesting developments we can (and have spoken about).
Both Sanchayan and I spoke a bit about our work with WebRTC-as-a-service provider, Daily. My talk at the GStreamer Conference was a summary of the work I wrote about previously about what we learned while building Daily's live streaming, recording, and other backend services. There were other clients we worked with during the year with similar experiences.
Sanchayan spoke about the interesting approach to building SIP support that we took for Daily. This was a pretty fun project, allowing us to build a modern server-side SIP client with GStreamer and SIP.js.
An ongoing project we are working on is building AES67 support using GStreamer for FreeSWITCH, which essentially allows bridging low-latency network audio equipment with existing SIP and related infrastructure.
As you might have noticed from previous sections, we are also working on a low-latency audio appliance using PipeWire.
Retrospective
All in all, we've had a reasonably productive 2023. There are things I know we can do better in our upstream efforts to help move merge requests and issues, and I hope to address this in 2024.
We have ideas for larger projects that we would like to take on. Some of these we might be able to find clients who would be willing to pay for. For the ideas that we think are useful but may not find any funding, we will continue to spend our spare time to push forward.
If you made this this far, thank you, and look out for more updates!
Editor's note: this work was completed in late 2022 but this post was unfortunately delayed.
Modern day audio hardware these days comes integrated with Digital Signal Processors integrated in SoCs and audio codecs. Processing compressed or encoded data in such DSPs results in power savings in comparison to carrying out such processing on the CPU.
+---------+ +---------+ +---------+
| CPU | ---> | DSP | ---> | Codec |
| | <--- | | <--- | |
+---------+ +---------+ +---------+
This post takes a look at how all this works.
For the last year and a half, we at Asymptotic have been working with the excellent team at Daily. I'd like to share a little bit about what we've learned.
Daily is a real time calling platform as a service. One standard feature that users have come to expect in their calls is the ability to record them, or to stream their conversations to a larger audience. This involves mixing together all the audio/video from each participant and then storing it, or streaming it live via YouTube, Twitch, or any other third-party service.
As you might expect, GStreamer is a good fit for building this kind of functionality, where we consume a bunch of RTP streams, composite/mix them, and then send them out to one or more external services (Amazon's S3 for recordings and HLS, or a third-party RTMP server).
I've written about how we implemented this feature elsewhere, but I'll summarise briefly.
This is a slightly longer post than usual, so grab a cup of your favourite beverage, or jump straight to the summary section for the tl;dr.
Recently I was working on a GStreamer
plugin
in Rust. The plugin basically rounds the corners of an incoming video,
something akin to the border-radius property in CSS. Below is how it looks
like when running on a video.
The GStreamer pipeline for the same:
gst-launch-1.0 filesrc location=~/Downloads/bunny.mp4 ! \
decodebin ! videoconvert ! \
roundedcorners border-radius-px=100 ! \
videoconvert ! gtksink
This was my first time working on a video plugin in GStreamer. Had a lot to
learn on how to use the BaseTransform class from GStreamer, among other
things. Without getting into the GStreamer specific details here, I basically
ran into a problem for which I needed to do some debugging for figuring out
what was going on in the internals of GStreamer.
Now, while I never had problems using GDB from the command line, but, the way I was using it earlier was just not good enough. I would start the pipeline, then attach gdb to a running process, place breakpoints by manually typing out the whole thing and then start. For one off debugging sessions, where may be you just want to quickly inspect the backtrace from a crash or may be look into a deadlock condition where your code hung, this could be fine. However, when you have to repeat this multiple times, do a source code change, compile and then check again, it becomes frustrating.
Let's look at how we can make this easier.
It's 2021, and while many things have changed for the better, they haven't changed enough, particularly in technology.
Right from the get-go, we knew we wanted to build a diverse company. Our founding team was balanced in gender terms. Our next hire was someone intimately familiar with the world we inhabited, and whom we knew through a Rust meetup that he organised.
We are determined to ensure that our hiring pipeline reflects the world we live in (rather than the world we work in). We've had a broad set of people vet our hiring page to ensure that the language is inclusive, and welcoming. We've reached out to folks we know and asked them to help us reach under-represented members in tech.
These are baby steps. We will continue doing our very best to build an inclusive, and welcoming space for everyone. We're acutely aware that gender is not the only axis of representation, and this is something we hope to address over time as well.
Here are some of the policies we've adopted to further our aim of building a diverse, and inclusive space. As we grow (slowly and sustainably), our intention is to continue to put progressive and employee-friendly policies in place.
Remote work
Offices are often designed around male-centric preferences. From the ergonomics of the workstation, to the temperature of the room, or the typical work schedule, professional spaces rarely account for differing priorities or perspectives. Remote work allows for more diverse participation.
Working hours
A full working week for us is 35 hours. We want to allow you the flexibility of scheduling your work day in a way that's most comfortable to you, while allowing for reasonable overlap with your team. We don't work weekends.
Conferences
We encourage and support both speaking at, and attending conferences, all over the world.
Paid menstrual leave
Taking nilenso's lead, we have adopted a no questions asked paid menstrual leave policy.
Paid leave
We currently operate with a flexible ("as needed") paid leave policy. We encourage people to regularly take time off. As we grow, we will likely introduce a more formal policy.
We started asymptotic 2½ years ago, and have been fortunate to have had a wild and busy ride so far. Nevertheless, it's probably high time we introduced ourselves! :-)
We are a small team of people who care about software freedom and take pride in their craft. Being tinkerers, we like working on low-level systems and are happiest when we're close to the metal.
We contribute to open source projects upstream, and help companies use them in products that are useful to people in the real world. This allows us to produce a virtuous cycle of improving the projects we maintain, while ensuring that they solve real world problems.
We want to live in a world where open source software is the default choice, and we aim to achieve that by making the projects we work on be the best option available.
Together, we're building a diverse, inclusive and sustainable company. This may mean choosing to be slow and deliberate while we grow. We're fine with that, because we believe that is the right path for us to achieve our goals while upholding the values we care about.
If you would like to join us on our journey, or would like to talk to us, please reach out!