Brenden Blanco's guest appearances

Brenden Blanco

OVS Orbit

Larry Peterson

Alex Ellis

Dave Hunt

Josh Bailey

Peter Phaal

Brian O'Connor

Jesse Gross

Greg Ferro

Nicholas Gray

Kyle Mestery

Chang Kim

Cloudbase Solutions

Dave Neary

Frederick Kautz

Simon Horman

OwlTail

      Brenden Blanco is one of the most prolific developers working on the IO Visor Project, a Linux Foundation      Collaborative Project.  Brenden was an employee at PLUMgrid, the startup behind IO      Visor, until it was acquired by VMware.           The interview begins with a history of the layers that stacked up to form      IO Visor.  The history begins with the publication of The      BSD Packet Filter: A New Architecture for User-Level Packet Capture      at the 1993 Winter USENIX Conference, in January 1993.  This paper      introduced BPF, short for Berkeley Packet Filter, for selecting packets      to be copied to userspace for analysis.  BPF was a register-based,      RISC-like virtual machine (analogous to the Java virtual machine).  In      Brenden's words:           Two of the major points really seemed very wise to me.  It must be      protocol-independent: the kernel should not have to be modified to add      more protocol support... It must be general: the instruction set should      be rich enough to handle unforeseen uses.  They have a few more points      about efficiency and generality, but those two first ones have really      ended up creating kind of a platform that has stood the test of time.           Safety was also critical:           Any operating system developer has an inherent distrust of the userspace,      and the system calls that they're defining need to be secure against      attack, as well as ignorance or bad programming.  The API they expose      lets a code be uploaded but the kernel has to protect itself against bad      code.  It doesn't allow loops, so you can't create an infinite loop in      this virtual machine, and it doesn't allow access to memory that's out of      bounds... That produces something that's limited, but safe.  You can't do      everything that a programmer would want, but you can do so efficiently.           BPF came to Linux during the development of the 2.5 release series, according to LWN.           A second fork of the history comes from academic research on operating      system extensibility, which was a theme that grew to prominence during      the 1990s with extensible research operating systems such as SPIN and      the Exokernel.      Many different approaches were proposed, including those based on safe      languages like Modula-3 and Java.  In addition, Brenden draws an analogy      between BPF and the development of instruction set-based GPUs.           Brenden says that the movement to software-defined networking has not      been able to speed up operating system evolution.  Part of the goal of IO      Visor is to help speed up this evolution, by allowing the networking      subsystem to evolve independent of the rest of the operating system.  In      Brenden's words:           Five years ago or so, when we started seeing a move toward      software-defined networking, the hope was that some of this paradigm      would change a little bit.  But what I've seen in some of the solutions      is that some of this actually isn't the case.  You have a movement of      network functionality from hardware to software but it's still locked      into kind of the operating system life cycle, which is maybe faster than      hardware but isn't in data centers where we've been trying to address      this, it's still slower than the applications change, so the use cases      still change faster than the infrastructure.           Around 2010, Alexei Starovoitov started bringing together these two      branches by extending BPF to form eBPF, which added numerous features      such as extending registers to 32 to 64 bits, increasing the number of      registers from 2 to 10, additional instructions, and the ability to jump      forward and backward (with some restrictions) and most importantly, the      ability to call a restricted set of kernel helper functions and the      ability to work with data structures (“maps”).  These changes made the      platform a better target for compiling high-level language code and      better able to interact with its environments.  In addition, compilers      for eBPF started being integrated into the kernel, to make eBPF execution      faster on key architectures.           Brenden introduces the BCC, or BPF Compiler      Collection, one of the primary sub-projects within IO Visor.  BCC      provides a set of tools for building software in high-level languages,      such as C and Python, into BPF program objects, loading those programs      into running kernels, and interacting with them once they are loaded.      BCC also includes a large suite of example programs.           BPF programs are loaded into a kernel by attaching them to “hook      points.”  Typically, the programs attached to a hook point are invoked      when some particular event occurs.  For example, in networking, a BPF      program might be invoked whenever a packet is received on a particular      device, or for performance monitoring a BPF program might be attached to      a “kprobe” point.           Brenden explains how to use BCC to load a BPF program into the kernel      using a 1-line Python program.  A more sophisticated program can retain      handles and use them to interact with the program at runtime.  Brenden      describes how restrictions on BPF are reflected in what a C programmer      can include in a program.  Keeping the in-kernel safety verifier simple      and correct is paramount, which tends to reduce the maximum complexity of      programs that can be loaded.  The verifier is a limiting factor that      continues to evolve to make BPF more useful over time.           Currently the Open vSwitch community is considering whether to replace      the use of the Open vSwitch kernel module by BPF.  Brenden is in favor of      the idea and offers some of his thoughts.  First, his experience at      PLUMgrid shows that BPF is flexible enough to support a wide variety of      network applications, including those that Open vSwitch implements.      Second, it's an enjoyable experience for a single developer to be able to      cover the entire infrastructure for an application.           In the future, Brenden is looking forward to BPF usage becoming      ubiquitous for Linux performance monitoring and other non-networking use      cases, such as storage acceleration and security.  BPF is already, for      example, used      for sandboxing in Chrome.  He's also looking forward to BPF for      hardware offload; it can already be used for hardware offload on      Netronome NICs.           For more information on IO Visor, please visit iovisor.org.  To talk to Brenden and      other IO Visor developers, visit the #iovisor channel on the oftc.net IRC      network.  Brenden's nick is bblanco.  You can also tweet to Brenden at @BrendenBlanco.           More about BPF:              In episode 11, John Fastabend from Intel talks about        BPF on network edge nodes.               In episode 4, Thomas Graf from Cisco talks about        Cilium, which uses BPF to address the question of how to address policy        in a legacy-free container environment that scales to millions of        endpoints.        Packet        Pushers PQ Show 60, from 2015, interviewed Pere Monclus from        PLUMgrid about the IO Visor Project and Linux networking.              OVS Orbit is produced by Ben Pfaff.  The      intro music in this episode is Drive,      featuring cdk and DarrylJ, copyright 2013, 2016 by Alex.  The bumper      music is Yeah Ant      featuring Wired Ant and Javolenus, copyright 2013 by Speck.  The outro      music is Space      Bazooka featuring Doxen Zsigmond, copyright 2013 by Kirkoid.  All      content is licensed under a Creative Commons Attribution 3.0      Unported (CC BY 3.0) license.    

Brenden Blanco is one of the most prolific developers working on the <a href="https://www.iovisor.org/" rel="nofollow" target="_blank">IO Visor Project</a>, a <a href="https://www.linuxfoundation.org/projects" rel="nofollow" target="_blank">Linux Foundation Collaborative Project</a>. Brenden was an employee at <a href="http://www.plumgrid.com/" rel="nofollow" target="_blank">PLUMgrid</a>, the startup behind IO Visor, until it was acquired by VMware. The interview begins with a history of the layers that stacked up to form IO Visor. The history begins with the publication of <a href="https://www.usenix.org/legacy/publications/library/proceedings/sd93/mccanne.pdf" rel="nofollow" target="_blank">The BSD Packet Filter: A New Architecture for User-Level Packet Capture</a> at the 1993 Winter USENIX Conference, in January 1993. This paper introduced BPF, short for Berkeley Packet Filter, for selecting packets to be copied to userspace for analysis. BPF was a register-based, RISC-like virtual machine (analogous to the Java virtual machine). In Brenden's words: <blockquote> Two of the major points really seemed very wise to me. It must be protocol-independent: the kernel should not have to be modified to add more protocol support... It must be general: the instruction set should be rich enough to handle unforeseen uses. They have a few more points about efficiency and generality, but those two first ones have really ended up creating kind of a platform that has stood the test of time. </blockquote> Safety was also critical: <blockquote> Any operating system developer has an inherent distrust of the userspace, and the system calls that they're defining need to be secure against attack, as well as ignorance or bad programming. The API they expose lets a code be uploaded but the kernel has to protect itself against bad code. It doesn't allow loops, so you can't create an infinite loop in this virtual machine, and it doesn't allow access to memory that's out of bounds... That produces something that's limited, but safe. You can't do everything that a programmer would want, but you can do so efficiently. </blockquote> BPF came to Linux during the development of the 2.5 release series, <a href="https://lwn.net/Articles/599755/" rel="nofollow" target="_blank">according to LWN</a>. A second fork of the history comes from academic research on operating system extensibility, which was a theme that grew to prominence during the 1990s with extensible research operating systems such as <a href="https://en.wikipedia.org/wiki/SPIN_(operating_system)" rel="nofollow" target="_blank">SPIN</a> and the <a href="https://en.wikipedia.org/wiki/Exokernel" rel="nofollow" target="_blank">Exokernel</a>. Many different approaches were proposed, including those based on safe languages like Modula-3 and Java. In addition, Brenden draws an analogy between BPF and the development of instruction set-based GPUs. Brenden says that the movement to software-defined networking has not been able to speed up operating system evolution. Part of the goal of IO Visor is to help speed up this evolution, by allowing the networking subsystem to evolve independent of the rest of the operating system. In Brenden's words: <blockquote> Five years ago or so, when we started seeing a move toward software-defined networking, the hope was that some of this paradigm would change a little bit. But what I've seen in some of the solutions is that some of this actually isn't the case. You have a movement of network functionality from hardware to software but it's still locked into kind of the operating system life cycle, which is maybe faster than hardware but isn't in data centers where we've been trying to address this, it's still slower than the applications change, so the use cases still change faster than the infrastructure. </blockquote> Around 2010, Alexei Starovoitov started bringing together these two branches by extending BPF to form eBPF, which added numerous features such as extending registers to 32 to 64 bits, increasing the number of registers from 2 to 10, additional instructions, and the ability to jump forward and backward (with some restrictions) and most importantly, the ability to call a restricted set of kernel helper functions and the ability to work with data structures (“maps”). These changes made the platform a better target for compiling high-level language code and better able to interact with its environments. In addition, compilers for eBPF started being integrated into the kernel, to make eBPF execution faster on key architectures. Brenden introduces the <a href="https://www.iovisor.org/technology/bcc" rel="nofollow" target="_blank">BCC</a>, or BPF Compiler Collection, one of the primary sub-projects within IO Visor. BCC provides a set of tools for building software in high-level languages, such as C and Python, into BPF program objects, loading those programs into running kernels, and interacting with them once they are loaded. BCC also includes a large suite of example programs. BPF programs are loaded into a kernel by attaching them to “hook points.” Typically, the programs attached to a hook point are invoked when some particular event occurs. For example, in networking, a BPF program might be invoked whenever a packet is received on a particular device, or for performance monitoring a BPF program might be attached to a “kprobe” point. Brenden explains how to use BCC to load a BPF program into the kernel using a 1-line Python program. A more sophisticated program can retain handles and use them to interact with the program at runtime. Brenden describes how restrictions on BPF are reflected in what a C programmer can include in a program. Keeping the in-kernel safety verifier simple and correct is paramount, which tends to reduce the maximum complexity of programs that can be loaded. The verifier is a limiting factor that continues to evolve to make BPF more useful over time. Currently the Open vSwitch community is considering whether to replace the use of the Open vSwitch kernel module by BPF. Brenden is in favor of the idea and offers some of his thoughts. First, his experience at PLUMgrid shows that BPF is flexible enough to support a wide variety of network applications, including those that Open vSwitch implements. Second, it's an enjoyable experience for a single developer to be able to cover the entire infrastructure for an application. In the future, Brenden is looking forward to BPF usage becoming ubiquitous for Linux performance monitoring and other non-networking use cases, such as storage acceleration and security. BPF is already, for example, <a href="http://blog.cr0.org/2012/09/introducing-chromes-next-generation.html" rel="nofollow" target="_blank">used for sandboxing in Chrome</a>. He's also looking forward to BPF for hardware offload; it can already be used for hardware offload on Netronome NICs. For more information on IO Visor, please visit <a href="https://www.iovisor.org" rel="nofollow" target="_blank">iovisor.org</a>. To talk to Brenden and other IO Visor developers, visit the #iovisor channel on the <a href="http://oftc.net" rel="nofollow" target="_blank">oftc.net</a> IRC network. Brenden's nick is bblanco. You can also tweet to Brenden at <a href="https://twitter.com/BrendenBlanco" rel="nofollow" target="_blank">@BrendenBlanco</a>. More about BPF: <ul> <li> In <a href="#e11" rel="nofollow" target="_blank">episode 11</a>, John Fastabend from Intel talks about BPF on network edge nodes. </li> <li> In <a href="#e4" rel="nofollow" target="_blank">episode 4</a>, Thomas Graf from Cisco talks about Cilium, which uses BPF to address the question of how to address policy in a legacy-free container environment that scales to millions of endpoints. </li> <li> <a href="http://packetpushers.net/podcast/podcasts/pq-show-60-io-visor-project-linux-networking/" rel="nofollow" target="_blank">Packet Pushers PQ Show 60</a>, from 2015, interviewed Pere Monclus from PLUMgrid about the IO Visor Project and Linux networking. </li> </ul> OVS Orbit is produced by <a href="mailto:blp@ovn.org">Ben Pfaff</a>. The intro music in this episode is <a href="http://dig.ccmixter.org/files/AlexBeroza/43098" rel="nofollow" target="_blank">Drive</a>, featuring cdk and DarrylJ, copyright 2013, 2016 by Alex. The bumper music is <a href="http://dig.ccmixter.org/files/speck/42100" rel="nofollow" target="_blank">Yeah Ant</a> featuring Wired Ant and Javolenus, copyright 2013 by Speck. The outro music is <a href="http://dig.ccmixter.org/files/Kirkoid/43005" rel="nofollow" target="_blank">Space Bazooka</a> featuring Doxen Zsigmond, copyright 2013 by Kirkoid. All content is licensed under a Creative Commons <a href="http://creativecommons.org/licenses/by/3.0/" rel="nofollow" target="_blank">Attribution 3.0 Unported (CC BY 3.0)</a> license.

The IO Visor Project, with Brenden Blanco from VMware