Mindfire Technology

A Short Explanation of Hierarchal Navigable Small Worlds (HNSW) Index for pgvector

Tue, 18 Jun 2024 09:00:00 -0600

In our last two posts (here and here) we first installed PostgreSQL and then installed the pgvector extension to add functionality to PostgreSQL to allow us to use PostgreSQL as our document store for use with Retrieval Augmented Generation (RAG) for use with Large Language Models (LLMs).

Before we continue on to use pgvector and PostgreSQL for RAG (similar to how we did in this post) we should first briefly explain how pgvector handles similarity comparisons. Moreover, how does it differ from our simple cosine similarity that we previously used as the basis for a semantic search (back in this post.)

Recall that a cosine similarity (as explained in this post) requires us to first embed every single passage then embed our query. Then we do a comparison of the embedded query to every single embedded passage. The result is a slow process. It can take minutes to do each query. This isn’t very useful for real life cases where we need a quick response to avoid losing our user’s interest. So how can we speed up the process?

One very clever answer is to use an algorithm called the Hierarchal Navigable Small World (HNSW) algorithm. The HNSW algorithm was first published in a 2016 paper called “Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs” by Malkov and Yashunin (found here).

This short post isn’t going to go into all the details of how the HNSW algorithm works in detail, but I’m going to give a high level ‘gist’ of what it does and why it significantly speeds up cosine similarity comparisons. I’ll also include some great links to posts that explain it in more detail.

Consider the way we did our semantic search back in this post. We loaded a copy of the Federalist Papers, split it up into chunks of text, and then embedded each chuck. When we did a query we had to compare to every single chuck of text using a straight linear search, i.e. starting with the first one and trying out every single one until we get to the end of the list. So, if we have 10,000 chunks of text to search through we must perform 10,000 cosine similarity calculations.

How might we utilize Artificial Intelligence algorithms to speed this search up?

To visualize this, imagine all the chunks of text existing on a graph. Except this graph is likely a 768-dimensional graph instead of a 2-D graph. But since we can’t imagine what 768 dimensions looks like, we’ll just imagine this in 2 dimensions instead, though the concept is identical.

Suppose the embedding for our query sits somewhere within this graph next to chunks of text. What we want is the chunk (or chunks) of text closest to the query.

There is an Artificial Intelligence (AI) algorithm that works something like that called the k-nearest neighbor (KNN) algorithm. In this example (taken from Wikipedia) the green circle could be thought of as our query and blue boxes and red triangles represent the chunks of text we’re comparing to.

So, in this example, we’d probably want to return either the top 3 closest chunks of text (in the solid circle) or the top 5 (in the dotted circle).

A lot of research has gone into how to speed up a KNN search. HNSW utilizes some of these clever speed ups to KNN so that it can quickly find the best documents without having to do a comparison with every single chunk of text.

Imagine that the chunks of text are connected to each other in a graph that likes something like this (taken from the 2016 HNSW paper):

You enter the graph where the red dotted line enters and then you traverse the graph until you find a node (i.e. the chunk of text) that has no connected nodes that are closer to your query. Because you don’t have to search every single node the process is quite a bit faster. The downside is that you may not find the closest match this way, but you should find something pretty close.

However, even with this speed up, you will quickly overwhelm the algorithm as the number of documents grows and the process will become intractable again.

So, imagine that you have multiple layers of graphs at different levels of abstraction. Something like this (taken from the 2016 paper)

You’d be able to search the top layer first (which is a very sparse graph, so there aren’t many nodes to search!), find the closest node, then descend a level to find a closer node, descend again, etc. The result would be a much faster search even if the number of documents got quite large.

This is in essence what the HNSW algorithm does. It creates layers of such graphs and keeps them properly connected to allow for a fast search. The downside is that the HNSW algorithm isn’t guaranteed to return the absolute closest matches.

pgvector creates a new kind of field (a vector) with a new kind of index that allows you to do a cosine similarity search using an HNSW index.

For now, that’s really all you need to know to understand how we’re going to utilize PostgreSQL and pgvector to do a faster cosine similarity search.

Links:

An excellent post explaining HNSW in detail: https://www.datastax.com/guides/hierarchical-navigable-small-worlds
An even more detailed explanation: https://towardsdatascience.com/similarity-search-part-4-hierarchical-navigable-small-world-hnsw-2aad4fe87d37
Interesting post about HNSW and pgvector: https://www.crunchydata.com/blog/hnsw-indexes-with-postgres-and-pgvector
The pgvector repo: https://github.com/pgvector/pgvector
Great post on HNSW and pgvector: https://neon.tech/blog/understanding-vector-search-and-hnsw-index-with-pgvector

Native Isn't Really Better

Fri, 25 Aug 2023 12:00:00 -0600

Old Rivalries

Mobile Application development is one of the most popular software specializations in the world, with an estimated 224,000 developers located in the United States alone. Nearly every major app or program has some mobile variant. You could probably think of a dozen mobile apps that you use regularly that are also present on your computer or tablet. Moreover, these apps are usually not exclusive to one type of mobile device, whether that means tablets and smartphones, or iOS and Android.

For those of us who are not overly familiar with the software development process for mobile apps, the lines between iOS and Android or even Windows and MacOS may seem blurred or unimportant. Yet, as those who have worked in the industry for many years know very well, the differences between these operating systems are drastic when it comes to software development and can completely reshape the way an engineer must design their code.

Operating systems, such as Android and iOS, run on entirely different languages known as native languages. This is similar to different countries with different cultures and languages, each with entirely different ways of accomplishing more-or-less the same tasks or end-goals. These coding languages can also be the defining aspect of a software engineer and their career, especially if they choose to specialize in a particular language rather than spread their talent across multiple languages.

Because of this, there has grown a divide between software engineers, each deciding for themselves which languages and tools are best to accomplish different tasks, or which languages and tools they are more comfortable using. For the last couple decades, since the rivalry between Microsoft and Apple began with their unique and yet similarly popular operating systems, the argument has mostly been between which native languages or tools to use, that of Microsoft’s .NET with C#, Google’s Java and Kotlin, or Apple’s Objective-C (or more recently, Swift). Moreover, there is also the argument of which preferred IDE (integrated development environment) to use, that of either Microsoft’s Visual Studio or Apple’s Xcode.

However, in the last decade another debate has enflamed: cross-platform tools, which build a bridge between different operating systems. The debate being less about which native language is best, and more about whether to go native and write unique code for each mobile operating system , or to use cross-platform tools which apply the written code to both iOS and Android simultaneously.

The Future: Cross-platform Frameworks

The idea is relatively simple in concept: a framework or tool that can translate code written from a single language to various other languages or platforms.

The upsides to something like this are easily understood, as now a software engineer who is specialized in only one type of language can now develop applications for multiple operating systems without having to learn an entirely new language. In opposition, however, there used to be many valid arguments against cross-platform such as issues with compatibility, latency, and build size- yet, in recent years that is no longer the case. In fact, it’s gotten to the point that cross-platform frameworks have started to make developing across multiple operating systems individually look downright wasteful and short sighted.

“.NET, that is Xamarin and MAUI, now compile down to pure native code, and they’ve been optimized to the point that once you’ve compiled your app and built it for the final target platform of Android or iOS, it’s going to run as fast- and sometimes faster- than native code would.” Says Tony Pitman, a software engineer with over 40 years of experience, who has watched this progression closely and has become an expert in the use of Xamarin and .NET MAUI. “Because, when you write native code, you’re the one doing all the work figuring out how to optimize it, and the .NET team has done a great job optimizing their code to produce extremely performant native code on Android and iOS.”

It would seem, then, that all the old arguments against cross-platform frameworks may no longer be relevant. It might, in rare cases, be more efficient for software developers to still use native code to write their apps, but only if they intend to release it for just the one OS and if their engineer is already specialized or trained in that native language. Otherwise, if you need to run your code/app on multiple operating systems, it may genuinely be a waste of money and time to write native code for each individual use case when the tools used for cross-platform development are near perfect replacements.

Consider the amount of time and resources wasted, all being spent on developing applications for different devices and operating systems. Not only are you doubling the time it takes to write the code, as you have to write it entirely from scratch per native language, but you are also, most likely, are paying an entirely different team of software developers trained on that particular language.

To make it even more difficult to justify, having code written in two different languages means you’ll end up with entirely different and separate codebases that will need to be maintained simultaneously. This means that if you have an app which is both on iOS and Android, and you need to make an update to the app, you’ll have to make the needed changes individually. The complication with this is not just twice the time and effort needed, but also the potential of complications with one OS/device causing a delay for both deployments. You might have that update ready to go on Android but be forced to delay its launch all because of an issue which is unique to iOS.

Just Like Mother Used to Make It

Now, to further elaborate on the point regarding how close cross platform frameworks get their code to native: these tools have been meticulously refined over the years to grapple into every part of a native application. For example, native components such as entry boxes (i.e., a fillable text form) can be very unique between operating systems, having their own special style that is reflective of that OS’ theme. In light of this, Xamarin and MAUI have been designed to cross-compile into those native components, meaning the same entry box will look as if it was written in native code on each device.

The value in this is that developers don’t have to spend extra effort ensuring that their apps fit the unique feeling of each operating system. The downside is that if you wanted the theme to be identical between devices and operating systems, then you have some additional work to do. The upside in turn, however, is that these cross-platform frameworks have tools and functionality to allow just that, including the ability to overwrite and/or adjust how those native components will appear after cross-compiling.

For example, on Android many text entry boxes (especially on websites) contain a magenta or purple underline rather than a box outline, while iOS usually has a recessed box with an outline. With Xamarin, you can choose to have your text entry element displayed how Android or iOS would natively display such a box as if you had written said app in their native language. Or, if you wanted both to simply show as an empty outlined box, you could adjust the custom properties to force it to display identically as such. Either way it is a one-time effort that can be executed using a language the developer knows, and the cross-platform framework will do the heavy lifting of compiling to both operating systems, seamlessly creating the desired component.

Options for every Developer

The majority of the arguments made to this point are specific to Xamarin and MAUI. Meaning, the ability to cross-compile into native components is (mostly) a unique feature to their platforms. Other solutions such as Flutter or ReactNative get around this by having their own unique components, which can be customized or adjusted according to the desires and needs of the developer.

This presents another interesting selling point of getting into the cross-platform framework arena, that there are multiple frameworks a developer can choose from. So, every developer has the opportunity to choose a platform that best suits their coding preferences.

“When it comes to choosing a cross-platform development tool, a lot of that decision is going to depend on the developer’s experience and what they’re used to. If you have a developer who is used to using Microsoft tools and writing Windows desktop apps and similar mobile apps, they’re more likely to use Xamarin or MAUI just because they’re used to .NET and they have a lot of experience with C#.” Says Tony, thinking back on his own experience with .NET and C#, which led him to Xamarin. “On the other hand, if you have a developer who is more web based, so they do websites and web pages, and a lot of JavaScript, they’ll probably gravitate more towards ReactNative or Flutter.”

He continues, elaborating on the flexibility of these tools, and how all that really matters is what you are comfortable with and what you are willing to learn, stating “Now that doesn’t mean that either one of those people couldn’t learn the other tool- in fact, I know a lot of developers who are Microsoft .NET people and yet use Flutter, and also people who are more web-based who really like .NET and C#.”

Conclusion

Ten years ago there could’ve been arguments made about the superiority of writing in native code with native tools due to the rather primitive cross-platform tools of the time. However, that is no longer the case, not even in the slightest. Not only can these tools compile applications to near identical replications of a natively written app but can even supersede them in terms of quality and performance. Most importantly, however, they can cut down development time significantly, reducing costs and allowing your developers to spend more time innovating rather than duplicating work and maintaining multiple codebases. It’s just plain better

What are your thoughts on the state and use of cross-platform development systems? Do you still prefer to use native languages and tools? Let us know your thoughts in the comment section below.