The Labs team shares their news, updates, and knowledge

Posts, articles and tutorials

blog

header

And how your Python library can become dataframe-agnostic too

How Narwhals and scikit-lego came together to achieve dataframe-agnosticism

An overview of the dataframe landscape, and solution to the "we only support pandas" problem

Dataframe interoperability - what has been achieved, and what comes next?

The journey of writing string ufuncs and creating the np.strings namespace for NumPy 2.0

Writing fast string ufuncs for NumPy 2.0

Presenting our 2023 annual report! Read about our open source project and community highlights, initiatives, and work culture.

Quansight Labs Annual Report 2023: Building for Ecosystem-wide Impact and Sustainability

What are those words on the bottom of your video screen and where do they come from? Captioning’s normalization in the past several decades may seem like it would render those questions moot, but understanding more about captions means making more informed decisions about when, how, and why we make sure information is accessible.

Captioning: A Newcomer’s Guide

A quick overview of the new Numba engine in DataFrame.apply

Unlocking C-level performance in pandas.DataFrame.apply with Numba

We are excited to spread the news about the improvements that have been taking place in CuPy, where 18 interpolation and more than 100 signal processing parallel GPU APIs are now available as part of a EOSS4 CZI grant.

Improving the interpolation and signal processing capabilities of CuPy

Moving SciPy to Meson meant finding a different Fortran compiler on Windows, which was particularly tricky to pull off for conda-forge. This blog tells the story about how things looked pretty grim for the Python 3.12 release, and how things ended up working out just in the nick of time.

The 'eu' in eucatastrophe – Why SciPy builds for Python 3.12 on Windows are a minor miracle


My work was focused on improving NumPy support in Numba, with focus on the polynomial package.

Adding support for polynomials to Numba

A journey through NumPy's Python API from a maintenance perspective.

Refining NumPy's Python API for its 2.0 release

SymPy's documentation has received many significant improvements over the past two years thanks to funding by the Chan Zuckerberg Initiative.

Improving SymPy's Documentation

Doctesting for PyData Libraries

Gives an introduction to the utility of hypothesis in SymPy

Integrating Hypothesis into SymPy

How can SciPy use the Array API Standard to achieve array library interoperability?

The Array API Standard in SciPy

A summary of my contributions to the Code-Generator project and PyTorch-Ignite ecosystem in the past few months as Quansight Labs intern and my learnings in the process.

Bridging Data Science Tools with PyTorch-Ignite's Code-Generator and Nebari

In this blog post, we share how scikit-learn enabled support for the Array API Standard.

Array API Support in scikit-learn

In this post I'm sharing my experience of traveling to the US for PyCon US 2023

PyCon US 2023 - An action-packed week

In the following blogpost, we will explore the newly added feature in Numba: Dynamic exception support. We will discuss the previous limitations and explain how Numba was enhanced to handle runtime exceptions.

Numba Dynamic Exceptions

Presenting our first annual report! Read about our project achievements, community initiatives, and work culture.

Quansight Labs Annual Report 2022: Celebrating Growth and Sustainability in Open Source

Potential solutions for pain points when dealing with native code; what needs unifying in the Python packaging space, and how should that be approached? 

Python packaging & workflows - where to next?

Blogpost of working on the PyTorch-Ignite project during internship at Quansight

Sangho's Internship at Quansight with PyTorch-Ignite project

Surbhi Sharma shares her exciting experience working as an intern at Quansight Labs and contributing to condacolab, a tool that lets you deploy a Miniconda installation easily on Google Colab notebooks. This enables you to use conda or mamba to install new packages on any Colab session.

Conda on Colaboratory

Kulsoom Zahra learns about accessibility and fixes a part of the JupyterLab interface (that used to break when zoomed in) during her summer 2022 internship at Quansight Labs.

Zoom zoom zoom! Improving Accessibility in JupyterLab

accessible-pygments hosts curated WCAG-compliant themes for all your syntax highlighting needs.

Making pygments accessible

In this blogpost, I share my experience as a Google Season of Docs 2022 technical writer working on updating the Editor user documentation.

The new Spyder Editor documentation under the spotlights!

Learning from awesome mentors and contributing to pandas open source

Close Encounter with pandas and the Jedis of open source

We are delighted to share details about new grants to support the sustainability of SciPy, conda-forge, and CuPy

Quansight Labs awarded three CZI EOSS Cycle 5 Grants

The Nebari CLI consists of various commands the user needs to run to initialize, deploy, configure, and update Nebari.

Developing a Typer CLI for Nebari

Quansight Labs is delighted to welcome its second cohort of 6 interns, who will work on a variety of open source projects and tasks

Introducing the 2022 Interns Cohort

Announcing the SciPy 2022 Accessibility Awareness Efforts

SciPy 2022 Accessibility Awareness Programs

A non-exhaustive but totally honest checklist for accessibility review

Checking for accessibility: thoughts and a checklist!

The development story of a developer command-line interface (CLI) for the SciPy project, with exmaples

The evolution of the SciPy developer CLI

In our weekly show and tell we got real about "why can writing blog  posts be so hard?" and collaboratively wrote up this blog post about what we learned  from the discussion.

Why is writing blog posts hard?

How we can use the Python Array API Standard with the fundamental libraries in the PyData ecosystem along with CuPy for making GPUs accessible to the users of these libraries

Making GPUs accessible to the PyData Ecosystem via the Array API Standard.

The Chan Zuckerberg Initiative has funded efforts to make the Jupyter ecosystem, starting with JupyterLab, more accessible. As a part of these increased efforts, the team will be providing a periodically updated list of what is currently being worked on and what is coming soon.

Jupyter accessibility efforts have a roadmap!

Grayskull is an automatic conda recipe generator, with a focus on conda-forge.

Conda and Grayskull, the Masters of Software Packaging

This is a companion post from the Official release of IPython 8.0. We hope it will help you apply best practices, and have an easier time maintaining your projects, or helping other.

IPython 8.0, Lessons learned maintaining software

A lot of us showed up for the code, but hung around for the community. We'll continue this post talking about the monthly Jupyter community calls, and how they help all jovyans, Project Jupyter's pet name for their developers and users, stay connected.

A year of Jupyter community calls

In this post, we aim to articulate that vision and suggest a path to making it concrete, focusing on three libraries at the core of the PyData ecosystem: SciPy, scikit-learn and scikit-image. 

A vision for extensibility to GPU & distributed support for SciPy, scikit-learn, scikit-image and beyond

My work was majorly focused on providing performance benchmarks to NumPy in realistic situations. The target was to show the world that NumPy is efficient in handling quasi real-life situations too.

NumPy Benchmarking

In the next lines, I"ll try to capture my experience at Quansight Labs as an intern working on the cuDF implementation of the dataframe interchange protocol. cuDF is a dataframe library very much like pandas which operates on the GPU in order to benefit from its computing power.

Dataframe interchange protocol: cuDF implementation

An exploration of a method to call C++ libarary function from Numba compiled functions, i.e. python function that are decorated with "numba.jit(nopython=True)"

An efficient method of calling C++ functions from numba using clang++/ctypes/rbc

Let's dive into the technicalities involved in array libraries interoperability and understand the protocols making this a reality.

Array Libraries Interoperability

In this blog post I talk about the projects and my work during my internship at Quansight Labs. My efforts were geared towards re-engineering CI/CD pipelines for SciPy to make them more efficient to use with GitHub Actions.

Re-Engineering CI/CD pipelines for SciPy

The work I briefly describe in this blog post is the implementation of the dataframe interchange protocol into Vaex which I was working on through the three month period as a Quansight Labs Intern.

Dataframe interchange protocol and Vaex

Low-code contribution workshops offer a way for folks to join a project's contributors list without the technical overhead. In this blog we present a workshop format that relies solely on the GitHub web interface.

Low-code contributions through GitHub

How we ended up with a discussion about where issues with JupyterLab’s built-in code editors fit on the list of immediate accessibility priorities.

Not a checklist: different accessibility needs in JupyterLab

A small team started working on making NumPy more accessible, specifically its website and documentation. They weren’t experts in accessible technology. In fact, they feared that they didn’t know enough. Yet they strongly believed that accessible technology is a right every person should access and that NumPy could be accessible too.

Making Numpy Accessible: Guidelines and Tools

We are thrilled to announce that the team at Quansight Labs has been awarded five EOSS Cycle 4 grants to work on several projects within the PyData ecosystem. This post will introduce the successful grantees and their objectives for these two-year long grants.

CZI EOSS4 Grants at Quansight Labs

Instead of going through all the complications involved in renting or acquiring dedicated hardware, setting up credentials and monitoring costs of a benchmarkin suite, we hoped we could use the same free cloud resources normally used for CI tests. Ideally, GitHub Actions.

Is GitHub Actions suitable for running benchmarks?

SciPy now builds with Meson on Linux, and the full test suite passes! This is a pretty exciting milestone, and good news for SciPy maintainers and contributors - they can look forward to much faster builds and a more pleasant development experience.

Moving SciPy to the Meson build system

PyTorch-Ignite is a practical solution, a high-level library from the PyTorch ecosystem for training neural networks. It is designed to simplify workflow development while maintaining maximum control, flexibility, and reproducibility. PyTorch-Ignite feels like a natural extension to PyTorch.

Introducing PyTorch-Ignite's Code Generator v0.2.0

In this blogpost we present pyflyby, a project and an extension to IPython and JupyterLab, that, among many things, automatically inserts imports and tidies Python files and notebooks.

Pyflyby: Improving Efficiency of Jupyter Interactive Sessions

Writing agnostic distributed code that supports different platforms, hardware configurations (GPUs, TPUs) and communication frameworks is tedious. In this blog, we will discuss how PyTorch-Ignite solves this problem with minimal code change.

Distributed Training Made Easy with PyTorch-Ignite

With the support of a team member with prior experience auditing for accessibility, we pinpointed specific ways in which JupyterLab lacked support for accessibility broken up by WCAG 2.1 standards.

Putting out the fire: Where do we start with accessibility in JupyterLab?

Today I want to look into a topic that has not evolved much since, and I believe could use an upgrade. Accessing interactive Documentation when in a Jupyter session, and what it could become. At the end I"ll link to my current prototype if you are adventurous.

Rethinking Jupyter Interactive Documentation

As a community manager in the Spyder team, I have been looking for ways of involving more users in the community and making Spyder useful for a larger number of people. With this, a new idea came: Education. For the past months, we have been wondering with the team whether Spyder could also serve as a teaching-learning platform, especially in this era where remote instruction has become necessary.

A step towards educating with Spyder

For contributors to the PyTorch codebase, one of the most commonly encountered C++ classes is TensorIterator. Recently, however, the interface has changed significantly. This post describes how to use the current interface as of April 2021.

PyTorch TensorIterator Internals - 2021 Update

Starting from Numba 0.53, Numba will ship with an enhanced version of the @guvectorize decorator. Similar to the @vectorize decorator, @guvectorize now has two modes of operation: Eager, or decoration-time compilation and Lazy, or call-time compilation

Enhancements to Numba's guvectorize decorator

This blog post contains the raw output of the 30-minute brainstorm (only cleaned up for textual issues) and my annotations on it (in italics) which capture some of the discussion during the session and links and context that may be helpful.

Python packaging in 2021 - pain points and bright spots

In this blog post we will focus specifically on recent improvements that have been made to SciPy's interpolation functions. A recent NumFOCUS small development grant awarded to the SciPy developers allowed a dedicated effort to fix existing bugs in boundary handling and improve the documentation of the behavior of these modes.

Making SciPy's Image Interpolation Consistent and Well Documented

Today I'm incredibly excited to welcome Tania Allard to Quansight as Co-Director of Quansight Labs. Tania is a well-known and prolific PyData community member. In the past few years she has been involved as a conference organizer (JupyterCon, SciPy, PyJamas, PyCon UK, PyCon LatAm, JuliaCon and more), as a community builder (PyLadies, NumFOCUS, RForwards), as a contributor to Matplotlib and Jupyter, and as a regular speaker and mentor. She also brings relevant experience in both industry and academia - she joins us from Microsoft where she was a senior developer advocate, and has a PhD in computational modelling.

Welcoming Tania Allard as Quansight Labs co-director

In the following blog post, we will show you step-by-step how you can develop a custom theme for JupyterLab, distribute it, and take the example of the jupyterlab-theme-winter theme we release today to celebrate the end of 2020.

Develop a JupyterLab Winter Theme

This new grant totaling $140,000 will fund part of our efforts to improve usability and sustainability in both projects and is excellent news for the scientific computing community, which will certainly benefit from this work downstream.

A second CZI grant for NumPy and OpenBLAS

This blog post is a conversation between two contributors, sharing their observations, experiences, and their hope for a stronger collaboration between design and development in open source.

Introduction to Design in Open Source

We'll highlight the ability of Ibis to generically prescribe query expressions across different data storage systems.

Querying multiple backends with Ibis

The basic installation format for users who install packages via pip is the wheel format. But what about Linux? The first manylinux standard, manylinux1, was based on CentOS5 which has been obsolete since March 2017. So what is next for manylinux, and what manylinux should users and package maintainers use?

Manylinux1 is obsolete, manylinux2010 is almost EOL, what is next?

In this post, I will detail some of the open source work that I have done recently, both as part of my open source consulting, and as part of my work on SymPy for Quansight Labs.

Quansight Labs Work Update for September, 2019

In this post, we'll go into detail on how the underlying design of the library works on a technical level. Versioned HDF5 is a library that wraps h5py and offers a versioned abstraction for HDF5 groups and datasets.

Design of the Versioned HDF5 Library

In this post, we'll show some of the performance analysis we did while developing the library, hopefully making the case that reading and writing versioned HDF5 files can be done with a nice, intuitive API while being as efficient as possible. The tests presented here show that using the Versioned HDF5 library results in reduced disk space usage, and further reductions in this area can be achieved with the use of HDF5/h5py-provided compression algorithms.

Performance of the Versioned HDF5 Library

You have probably seen Traitlets in applications, you likely even use it. The package has nearly 5 million downloads on conda-forge alone. But, what is Traitlets? In this post we'll answer this question along with where Traitlets came from, its applications, and a bit of history.

Traitlets - an introduction & use in Jupyter configuration management

Starting with IPython 7.16.1 (released in June 2020), you should be able to recreate the sdist (.tar.gz) and wheel (.whl), and get byte for byte identical result to the wheels published on PyPI. This is a critical step toward being able to trust your computing platforms, and a key component to improve efficiency of build and packaging platforms. It also potentially impacts fast conda environment creation for users. The following goes into some reasons for why you should care.

IPython reproducible builds

HDF5 is an open technology that implements a hierarchical structure (similar to a file-system structure) for storing large amounts of possibly heterogeneous data within a single binary file, using regular version control tools (such as git) may prove difficult. The Versioned HDF5 library is a versioned abstraction on top of h5py, that allows you to keep a record of which changes occurred to your HDF5 files, and enables you to recover previous versions of this file.

Introducing Versioned HDF5

Open source is notorious for lack of design presence, enough so that my search to prove this fact has turned up nearly nothing. Once I learned I shouldn’t expect that everyone on my team knows what they want from the design process, here’s what helped me move forward and collaborate with developers successfully.

Designing with and for developers

I joined Quansight at the beginning of April, splitting my time between PyTorch (as part of a larger Quansight team) and contributing to Quansight Labs supported community-driven projects in the Python scientific and data science software stack, primarily to NumPy.  I have found my next home; the people, the projects, and the atmosphere are an all around win-win for me and (I hope) for the projects to which I contribute.

Quansight Labs: what I learned in my first 3 months

I recently moved, and my new place has a relatively small footprint. (Yes, I moved during the COVID-19 pandemic. And yes, it was crazy.) I quickly realized that was going to need a wireless router of some sort, or more formally, a wireless access point (WAP).

Learn NixOS by turning a Raspberry Pi into a Wireless Router

At first, I didn’t actually understand how important docs are for software, especially for open source projects. However, during all this time I’ve learned how documentation has a huge impact on the open-source community and I’ve been thankful to have been able to do this. But, from the beginning, I asked myself “why am I the ‘right person’ for this?”

Writing docs is not just writing docs

A demonstration of a method to create a copy an exact dev environment on
one machine, and transfering it to a another machine.


Creating a Portable Python Environment from Imports

After more than three years in development and more than 5000 commits from 60 authors around the world, Spyder 4 finally saw the light on December 5, 2019!

Thanking the people behind Spyder 4

This post is a deep dive into how TensorIterator works, and is an essential part of learning to contribute to the PyTorch codebase since iterations over tensors in the C++ codebase are extremely commonplace. This post is aimed at someone who wants to contribute to PyTorch

PyTorch TensorIterator Internals

ndindex is a new library that provides high level objects representing the various objects that can index NumPy arrays. These objects automatically canonicalize under the assumption of NumPy indexing semantics, and can be manipulated with a uniform API. All ndindex types have a .args that can be used to access the arguments used to create the object, and they are all hashable.

Introducing ndindex, a Python library for manipulating indices of ndarrays

Having official high-level documentation written using up-to-date content and techniques will certainly mean more users (and developers/contributors) are involved in the NumPy community. So, if everybody agrees on its importance, why is it so hard to write good documentation?

Documentation as a way to build Community

We're quite excited by the number of students who have shown an interest in participating, and we look forward to seeing excellent applications! What's more exciting, though, are some of the first contributions from people not currently at Quansight, in the true spirit of open-source software!

uarray: GSoC Participation

An update about recent research activities and upcoming changes

Planned architectural work for PyData/Sparse

The Spyder-Terminal project is revitalized! The new 0.3.0 version adds numerous features that improves the user experience, and enhances compatibility with the latest Spyder 4 release, in part thanks to the improvements made in the xterm.js project.

Creating the ultimate terminal experience in Spyder 4 with Spyder-Terminal

This was my first time to present metadsl, although really a year earlier Travis and I co-presented on a previous incarnation of the project. I think once I got to the actual code example though, I could have done a better job actually explaining how the system worked and what its limitations are. But it did give me leaping off point to have many great conversations with folks afterwards, which I will summarize below.

metadsl PyData talk

In this post, we will be talking about the improvements made to the Variable Explorer. These include the brand new Object Explorer for inspecting arbitrary Python variables, full support for MultiIndex dataframes with multiple dimensions, and the ability to filter and search for variables by name and type, and much more.

Variable Explorer improvements in Spyder 4

I'm very pleased to announce that NumPy and OpenBLAS just received a $195,000 grant from the Chan Zuckerberg Initiative, through its Essential Open Source Software for Science (EOSS) program!

A new grant for NumPy and OpenBLAS!

We will focus on the Files pane in this post, where we've made several improvements to the interface and file management tools.

File management improvements in Spyder4

There comes a time in every project where most technological hurdles have been surpassed, and its adoption is a social problem. I believe uarray and unumpy had reached such a state, a month ago. I'd like to present some improvements the Quansight team has made to uarray over the course of the past several months.

uarray: Attempting to move the ecosystem forward

This post is a rather long and detailed introduction to the XND ruby wrapper. There will also be some details on the implementation of the wrapper and how it differs from the Python wrapper (which existed before the Ruby wrapper).

Ruby wrappers for the XND project

Through a community work order (CWO) with the D. E. Shaw group, the Quansight Labs team has been able to dedicate developer time towards bug fixes and feature requests for Dask.

Quansight Labs Dask Update

As part of our next release, we are proud to announce an additional completion client for Spyder, Kite. Kite is a novel completion client that uses Machine Learning techniques to find and predict the best autocompletion for a given text.

Spyder 4.0 beta4: Kite integration is here

This post is just a quick update to summarize Quansight's presence and contributions, as well as some of the more interesting things I noticed.

Quansight presence at SciPy'19

TDK-Micronas is sponsoring Spyder development efforts through Quansight Labs. This will enable the development of some features that have been requested by our users, as well as new features that will help TDK develop custom Spyder plugins in order to complement their Automatic Test Equipment (ATE’s) in the development of their Application Specific Integrated Circuits (ASIC’s).

TDK-Micronas partners with Quansight to sponsor Spyder

Time flies when you're having fun. Here is an update of some of the highlights of my second month at Quansight Labs.

Labs update and May highlights

Quansight Labs is an experiment for us in a way. One of our main aims is to channel more resources into community-driven PyData projects, to keep them healthy and accelerate their development. And do so in a way that projects themselves stay in charge.

Community-driven open source and funded development

Developers of open source software often have a difficult time understanding how others utilize their libraries. Having better data of when and how functions are being used has many benefits.

Measuring API usage for popular numerical and scientific libraries

This is the first of a series of blog posts we will be writing to showcase updates on the development of Spyder, new planned features and news on the road to Spyder 4.0 and beyond.