Mindfire Technology

How to Install Node.js (for Windows)

Tue, 04 Nov 2025 12:00:00 -0700

I’m going to explore n8n in future blog posts, so, go over how to set up an AI workflow. To keep this consistent with our ‘open sourced’ approach to AI, we’re going to do a self-hosted version of n8n using the free community edition. To install n8n you need to have Node.js already installed so that you can use the npm command. So, for this blog post we’re going to briefly go over how to install Node.js. This one is particularly easy to do and likely you already have it installed for some other purpose. But for completeness, let’s quickly cover it:

Update: It's actually recommended that you use tools like nvs (Windows) and nvm (Mac & Linux)

Additionally, there are very few (if any) packages that should be installed globally. Installing those as local dependencies can make a huge difference in your ability to move between projects that have different node versions and requirements. You can do this with the command parameter --save-dev.

Example:

>nvs use 20.15
>node --version
v20.15.0
>npm install typescript --save-dev

First, go to the node.js home page:

https://nodejs.org/

You’ll see this page below. Click on the “Get Node.js” button:

That will take you to the download page (or you can just click the link below and go directly there):

https://nodejs.org/en/download

The download page offers you several different ways to download Node.js. Since we’re a Windows shop (and since Windows needs more AI love) I’m going to show you how to download the Windows installer, though feel free to change this to whatever operating system you prefer. First you need to select the operating system of your choice, which for me is Windows 64-bit:

Then you click the “Windows Installer.msi” button to do the download:

After the download, click the browser download icon and select what you just downloaded and run it:

This starts the installer:

Now just take the defaults in the installer and you’re ready for the next step for installing n8n.

Google Gemini with Gradio

Wed, 26 Mar 2025 09:00:00 -0600

Gradio is an amazing little library that allows you to quickly build a UI for AI Chatbots or other AI related projects. Better yet, you can then either run that app locally or share it out on the web with Gradio hosting it for you. It can also be used with Hugging Face Spaces which will host your Gradio app for free. You’ll be able to show it off to clients! (You can find instructions for Hugging Face Space here and an overview here. But I’m planning to do a future blog post on this useful little tool.)

The Quick Start Guide

Getting Started with Gradio is pretty easy. This quick start guide is the right place to start. Let’s start with installing Gradio:

pip install --upgrade gradio

You are welcome to follow the short tutorial in the quick start guide. I actually found that useful. But let’s do something a bit more difficult for this post. We’re going to write a Gradio chatbot that uses Google Gemini’s API. Here is a well-done tutorial on how to integrate Gemini and Gradio.

It’s going to be a Dungeon Master chatbot that plays role playing games with you. It will look like this when it is working:

Getting Started with Google Gemini

We’ve had blog posts in the past about how to get started with Google Gemini. (Part 1 and Part 2. Plus how to integrate with Haystack.) However, things change fast! So, I’m going to quickly go over the current way to do it, but you may want to look at the past blog posts for a more detailed look.

First, navigate to the Gemini Quick Start Guide for Python. Then install Gemini:

pip install -q -U google-genai

You will need an API key which you can find in Google AI Studio, or you can click here to get instructions.

BTW, curious about Google AI Studio? Check out this article on it.

Creating a Gradio Chatbot

There are some basic instructions for how to create a Gradio chatbot found here. But let me just go over the basics of how to use Gradio with Gemini.

First, we’ll need a way to load the Gemini secret. I like this approach which I’ve used in past posts:

def get_secret(secret_file: str) -> str:
    try:
        with open(secret_file, 'r') as file:
            secret_text: str = file.read().strip()
    except FileNotFoundError:
        print(f"The file '{secret_file}' does not exist.")
        secret_text = ""
    except Exception as e:
        print(f"An error occurred: {e}")
        raise e

    return secret_text

Just use that function to read a secret out of a text file.

Next, we’ll need some imports:

import gradio as gr
from google import genai
from google.genai import types

Now we need to create the actual Gemini chatbot that we’re going to use. First, I get the Gemini API key (secret) out of my text file. (Never check in a secret into your repo!) Then I use it to login to Gemini and create a client.

google_secret: str = get_secret(r'D:\Documents\Secrets\gemini_secret.txt') 
client = genai.Client(api_key=google_secret)

How to Make a Dungeon Master

The next line is interesting.

config = types.GenerateContentConfig(system_instruction="You are a Dungeon Master that will play a game with me.")

I create a ‘config’ variable that setup a ‘system instruction’ which is instructions that get repeated with each chat exchange so that the chatbot doesn’t wander away from its intended purpose.

In this case, I instruct it to be a Dungeon Master. Keep in mind that the system instructions get repeated for each chat exchange, so it is best to keep it short.

Finally, I create the actual chatbot using gemini-1.5-flash.

chat = client.chats.create(model="gemini-1.5-flash", config=config)

The Gradio Response Function

Now we’re finally ready to create the ‘response’ function that Gradio always uses. By default, it has two parameters: message and history. ‘message’ is the current message the user typed and ‘history’ is the chat history which (as mentioned above) is a list of tuples that contain user messages and the chatbot’s response.

def response(message, history):
    global chat
    chat_response = chat.send_message(message)

    # Each character of the answer is displayed
    for i in range(len(chat_response.text)):
        yield chat_response.text[: i+1]

Notice that we don’t even use the history. Why? Because the Gemini ‘send_message” method automatically keeps a history. The loop that follows streams the response.

Finally, we need a way to launch the Gradio interface itself:

def main():
    demo = gr.ChatInterface(response,
                            title='RPG Chat',
                            textbox=gr.Textbox(placeholder="Chat to the Dungeon Master"),
                            )
    demo.launch(debug=True)


if __name__ == "__main__":
    main()

gr.ChatInterface creates the demo UI which we then ‘launch’. You can find the final copy of the code here. I added it to the Book Search Archive even though it isn’t integrated yet. I’ll do a post on that once I create a Gradio interface for the Book Search Archive.

Now run this program and you’ll see something like this:

Now run the given localhost link and you’ll see the interface:

Note how it said that if you call launch like this with Share set to True:

demo.launch(share=True)

The demo will instead launch hosted by Gradio and will be available publicly. Unfortunately, I couldn’t get that to work properly with my Gemini code because Microsoft Defender was not a fan of it. I decided it was beyond the scope of this blog post to work out those issues. But the basic idea is that you can theoretically run this Gradio chatbot hosted by Gradio so it’s available on the web.

Conclusions

We are just scratching the surface of how to utilize Gradio as a chatbot / AI interface. And we haven’t even gotten to how to use it with Hugging Face spaces. But this is an exciting open-source library that lets you quickly put together an amazing looking AI demo. In this post, we covered how to integrate Gradio’s UI with Google Gemini’s API.

Be sure to comment down below what you think of Gradio and how you could use it in your future projects.

Pulling It All Together: Docling for Loading PDFs

Wed, 19 Mar 2025 01:00:00 -0600

Up to this point I’ve written a number of blog posts on the use of Docling to load PDFs. Below is the list of relevant blog posts plus the functions and methods (covered under each post) that we’ll be using in this post.

Using Docling for PDF to Markdown
Using NLTK to remove dashes and hyphens
- is_valid_word()
- combine_hyphenated_words()
Using Docling to parse the PDF based on Docling created text labels
- _get_processed_texts()
- is_section_header()
- is_page_footer()
- is_page_header()
- is_footnote()
- is_text_break()
- is_page_not_text()
- is_page_text()
- is_ends_with_punctuation()
- is_text_item()
- is_bottom_note()
Using Docling to fix paragraphs broken across page boundaries
- is_sentence_end() <part 3>
- is_ends_with_punctionation() <part 3>
- combine_paragraphs() <part 3>

Now we’re going to pull everything we’ve done together for a Docling parser that is comparable to our EPub/HTML Parser, complete with metadata such as section names. This post will cover the rest of what you need to know.

A Few More Methods

Here is the commit at the time of writing this post so that you can follow along with the actual code.

I’m going to define a few more useful functions first:

First, a function to get the next text in the list. Think of it as looking ahead one text item so that we can make choices based on the label (e.g. ‘section header’) of that text:

def get_next_text(texts: List[Union[SectionHeaderItem, ListItem, TextItem]], i: int) \
        -> Optional[Union[ListItem, TextItem]]:
    # Seek through the list of texts to find the next text item using is_text_item
    # Should return None if no more text items are found
    for j in range(i + 1, len(texts)):
        if j < len(texts) and is_text_item(texts[j]):
            return texts[j]
    return None

Now let’s create some functions to clean up text:

def remove_extra_whitespace(text: str) -> str:
    # Remove extra whitespace in the middle of the text
    return ' '.join(text.split())


def clean_text(p_str: str) -> str:
    p_str = str(p_str).strip()  # Convert text to a string and remove leading/trailing whitespace
    p_str = p_str.encode('utf-8').decode('utf-8')
    p_str = re.sub(r'\s+', ' ', p_str).strip()  # Replace multiple whitespace with single space
    p_str = re.sub(r"([.!?]) '", r"\1'", p_str)  # Remove the space between punctuation (.!?) and '
    p_str = re.sub(r'([.!?]) "', r'\1"', p_str)  # Remove the space between punctuation (.!?) and "
    p_str = re.sub(r'\s+\)', ')', p_str)  # Remove whitespace before a closing parenthesis
    p_str = re.sub(r'\s+]', ']', p_str)  # Remove whitespace before a closing square bracket
    p_str = re.sub(r'\s+}', '}', p_str)  # Remove whitespace before a closing curly brace
    p_str = re.sub(r'\s+,', ',', p_str)  # Remove whitespace before a comma
    p_str = re.sub(r'\(\s+', '(', p_str)  # Remove whitespace after an opening parenthesis
    p_str = re.sub(r'\[\s+', '[', p_str)  # Remove whitespace after an opening square bracket
    p_str = re.sub(r'\{\s+', '{', p_str)  # Remove whitespace after an opening curly brace
    p_str = re.sub(r'(?<=\s)\.([a-zA-Z])', r'\1',
                   p_str)  # Remove a period that follows a whitespace and comes before a letter
    p_str = re.sub(r'\s+\.', '.', p_str)  # Remove any whitespace before a period
    # Remove footnote numbers at end of a sentence. Check for a digit at the end and drop it
    # until there are no more digits or the sentence is now a valid end of a sentence.
    while p_str and p_str[-1].isdigit() and not is_sentence_end(p_str):
        p_str = p_str[:-1].strip()
    return p_str

Without going into too much detail, the first removes extra spaces. So word1<space, space>word2 becomes word1word2. This isn’t strictly necessary, but it helps a lot in debugging when I’m trying to search for a phrase and my eyes can’t tell that there is an extra space there.

The second function is a series of regexes that clean up the text in a variety of ways. See the inline comments to see what they do.

Because many section titles start with a roman numeral, it’s often helpful to be able to detect if a word is really a roman numeral:

def is_roman_numeral(s: str) -> bool:
    roman_numeral_pattern = r'(?i)^(M{0,3})(CM|CD|D?C{0,3})(XC|XL|L?X{0,3})(IX|IV|V?I{0,3})$'
    return bool(re.match(roman_numeral_pattern, s.strip()))


Docling tracks page numbers, so let’s write a function to read the page number:
def get_current_page(text: Union[SectionHeaderItem, ListItem, TextItem],
                     combined_paragraph: str,
                     current_page: Optional[int]) -> Optional[int]:
    return text.prov[0].page_no if current_page is None or combined_paragraph == "" else current_page

Here is a helper function that checks if this is an element (based on Doclings label or if it is a roman numeral) that we don’t want to include in our document fragments that we’ll do a semantic search over:

def should_skip_element(text: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    return any([
        is_page_footer(text),
        is_page_header(text),
        is_roman_numeral(text.text)
    ])

At Last: The Main Loop!

Finally, we’re ready to go over the (now fairly simple) DoclingParser class I wrote that does all the magic. First let’s declare the class and the constructor:

class DoclingParser:
    def __init__(self, doc: DoclingDocument,
                 meta_data: dict[str, str],
                 min_paragraph_size: int = 300,
                 start_page: Optional[int] = None,
                 end_page: Optional[int] = None,
                 double_notes: bool = False):
        self._doc: DoclingDocument = doc
        self._min_paragraph_size: int = min_paragraph_size
        self._docs_list: List[ByteStream] = []
        self._meta_list: List[Dict[str, str]] = []
        self._meta_data: dict[str, str] = meta_data
        self._start_page: Optional[int] = start_page
        self._end_page: Optional[int] = end_page
        self._double_notes: bool = double_notes

And let’s create a final helper method to append a document fragment and its associated metadata into our growing list:

    def _add_paragraph(self, text: str, para_num: int, section: str,
                       page: Optional[int], docs: List[ByteStream], meta: List[Dict]):
        docs.append(ByteStream(text.encode('utf-8')))
        meta.append({
            **self._meta_data,
            "paragraph_#": str(para_num),
            "section_name": section,
            "page_#": str(page)
        })

And now the ‘run()’ method should make perfect sense:

def run(self) -> Tuple[List[ByteStream], List[Dict[str, str]]]:
    temp_docs: List[ByteStream] = []
    temp_meta: List[Dict[str, str]] = []
    combined_paragraph: str = ""
    i: int
    combined_chars: int = 0
    para_num: int = 0
    section_name: str = ""
    page_no: Optional[int] = None
    first_note: bool = False

    texts = self._get_processed_texts()

    for i, text in enumerate(texts):
        next_text = get_next_text(texts, i)
        page_no = get_current_page(text, combined_paragraph, page_no)

        # Check if the current page is within the valid range
        if self._start_page is not None and page_no is not None and page_no < self._start_page:
            page_no = None
            continue
        if self._end_page is not None and page_no is not None and page_no > self._end_page:
            if self._double_notes and not first_note:
                self._min_paragraph_size *= 2
                first_note = True
            continue

        # Update section header if the element is a section header
        if is_section_header(text):
            section_name = text.text
            continue

        if should_skip_element(text):
            continue

        p_str = clean_text(text.text)
        p_str_chars = len(p_str)

        # If the paragraph does not end with final punctuation, accumulate it
        if not is_sentence_end(p_str):
            combined_paragraph = combine_paragraphs(combined_paragraph, p_str)
            combined_chars += p_str_chars
            continue

        # p_str ends with a sentence end; decide whether to process or accumulate it
        total_chars = combined_chars + p_str_chars
        if is_section_header(next_text):
            # Immediately process if the next text is a section header
            p_str = combine_paragraphs(combined_paragraph, p_str)
            combined_paragraph, combined_chars = "", 0
        elif total_chars < self._min_paragraph_size:
            # Not enough characters accumulated yet; decide based on next_text
            if next_text is None or (not is_page_text(next_text) and is_sentence_end(p_str)):
                # End of document or next text item is not a text item and current paragraph ends with punctuation
                # Process the paragraph and reset the accumulator even though this is a short paragraph
                p_str = combine_paragraphs(combined_paragraph, p_str)
                combined_paragraph, combined_chars = "", 0
            else:
                # Combine with next paragraph
                combined_paragraph = combine_paragraphs(combined_paragraph, p_str)
                combined_chars = total_chars
                continue
        else:
            # Sufficient characters: process the paragraph and reset the accumulator
            p_str = combine_paragraphs(combined_paragraph, p_str)
            combined_paragraph, combined_chars = "", 0

        p_str = combine_hyphenated_words(p_str)
        if p_str:  # Only add non-empty content
            para_num += 1
            self._add_paragraph(p_str, para_num, section_name, page_no, temp_docs, temp_meta)
            page_no = None

    return temp_docs, temp_meta

Let’s break this down. First a loop:

    for i, text in enumerate(texts):
        next_text = get_next_text(texts, i)
        page_no = get_current_page(text, combined_paragraph, page_no)

Skip the page if it falls outside the valid range (as determined by the associated valid pages csv):

        # Check if the current page is within the valid range
        if self._start_page is not None and page_no is not None and page_no < self._start_page:
            page_no = None
            continue
        if self._end_page is not None and page_no is not None and page_no > self._end_page:
            if self._double_notes and not first_note:
                self._min_paragraph_size *= 2
                first_note = True
            continue

Is this text we can skip?

        # Update section header if the element is a section header
        if is_section_header(text):
            section_name = text.text
            continue

        if should_skip_element(text):
            continue

Clean the text:

        p_str = clean_text(text.text)
        p_str_chars = len(p_str)

The rest of the code is the real magic. It makes a decision about if we are going to make this paragraph a document fragment or try to combine it with another one. We would try to combine it either because it is too short on its own (i.e. we don’t want paragraphs that are too short to dominate the semantic search) or because it’s not really a paragraph due to, say, a page break. Here is the basic logic:

Check for the end of a sentence. If so, this is now (possibly combined with a previous paragraph) now a full paragraph we’ll save as a document fragment.
If the next text is a section header, then we want to accumulate the current paragraph and try to combine it with the next one (since we already know it is not a sentence end due to condition 1.
If it’s too small (smaller than given minimize size for a paragraph), accumulate it to combine with the next one.

Finally, we save the accumulated paragraph off with its metadata and move to the next text.

Conclusion

And that’s it! We now have a way to parse a PDF, clean up the text, and provide metadata such as section headers.

There is undoubtedly more to do here. For example, PDFs often read words wrong or places spaces between letters of a single word. I haven’t addressed deeper problems like this yet. Though I plan to do this in the future, this will probably require a lot more processing or possibly even using an LLM or other language model to figure out contextually what the erroneous word was meant to be.

And why to go through this much trouble? I mean isn’t it true that the semantic search will probably mostly figure it out anyhow?

Ah, but this isn’t just about doing a better semantic search (though that is part of it.) It’s also because we’re going to explore how to read the text via text-to-speech (TTS). And to really do a good job of that you definitely need to fix all the words or they will get mispronounced. But that is the subject of a future post.

Also, if you are interested in learning how Docling and our other tutorials on AI can be put to use within your business, reach out over our Contact-Us page for a free consultation and/or discussion on AI.

Finding Paragraphs in PDFs - Using IBM’s Docling

Tue, 11 Mar 2025 12:00:00 -0600

In a past post, I argued that chunking text by paragraphs in a book is likely a good idea because it means a human has already grouped the text in a way that is topically relevant. After all, a paragraph is a set of sentences that are topically grouped.

In fact, I’ve found that most human-created paragraphs in a book fit neatly into an ideal-sized context window for a text embedder. (This paper on the best open-sourced RAG stack suggests this context window should be no more than 512 tokens.)

On top of that, I’ve already covered how to handle paragraphs that are too large to fit into the embedder’s context window by chunking them in a way that guarantees they will fit.

However, one could argue that human chunking (i.e., paragraphs) isn’t the best approach. There are surely more scientific methods, like measuring semantic similarity between chunks through trial and error until we find the ideal chunks (i.e., semantic chunking).

But there’s another reason I prefer to keep human-created paragraphs together as a single unit: When I feed the text back to a human to read, it’s just easier for them to receive a full paragraph from a book rather than one that’s chunked up mid-paragraph or across paragraphs.

But that only works if we have a way to read paragraph by paragraph from a document. We had that with EPUB files, thanks to them being HTML and using the <p> tag for each paragraph, which made it trivially easy to read one paragraph at a time. But what about PDF documents?

As discussed in our previous post, this is not an easy feat. IBM’s Docling can help—it at least attempts to break the text into paragraphs. But it falls short at times. What we’ll need is a program that can pull broken paragraphs back together. But how might that work?

Removing Interrupting Text

We’ve already got two parts of the puzzle in place:

One thing I covered in the last post, but didn’t make explicit, was that I removed footnotes and bottom notes from the PDF’s text. I then inserted them back at the end to avoid losing them. This way, those interrupting texts are out of the way of the main text. By doing this, they no longer break up the text flow! (Yet we can still embed them into the document store.)

This is the first step towards bringing paragraphs back together.

Finding Ends of Sentences

The next trick is to be able to determine if a TextItem that Docling feeds us terminates correctly with the end of a sentence such as a period, question mark, or exclamation point. Let’s write some code to determine that: (All code examples found in this commit at time of writing this blog post.)

def is_sentence_end(text: str) -> bool:
    has_end_punctuation: bool = is_ends_with_punctuation(text)
    # Does it end with a closing bracket, quote, etc.?
    ends_with_bracket: bool = (text.endswith(")")
                               or text.endswith("]")
                               or text.endswith("}")
                               or text.endswith("\"")
                               or text.endswith("\'"))
    return (has_end_punctuation or
            (ends_with_bracket and is_ends_with_punctuation(text[0:-1])))

def is_ends_with_punctuation(text: str) -> bool:
    return text.endswith(".") or text.endswith("?") or text.endswith("!")

This method takes a text input and checks for a series of terminations that it considers a proper sentence ending. The is_ends_with_punctuation method simply checks for a period, question mark, or exclamation point. However, my testing showed that this alone often wasn’t reliable.

To improve accuracy, I use this method inside is_sentence_end, which also checks for sentence endings inside quotes, brackets, and similar cases. Together, these two methods work pretty well. Now, we have a way to detect if a text item ends with a sentence. This doesn’t guarantee we’ll always find the end of a paragraph—since a paragraph might break across two pages while still ending with a sentence—but it should get us pretty close.

I’ve wondered if NLTK has something built in to do the same. I need to explore that further. But I didn’t think of that until after I wrote this code and debugged it.

Next, we need a method to combine paragraphs together:

def combine_paragraphs(p1_str: str, p2_str: str):
    # If the paragraph ends without final punctuation, combine it with the next paragraph
    if is_sentence_end(p1_str):
        return p1_str + "\n" + p2_str
    else:
        return p1_str + " " + p2_str

Note that I use this both for combining parts of a single paragraph or combining two short paragraphs. That is why it checks if is_sntence_end and if so, it puts a new line between them (because they are two distinct short paragraphs I want to combine). Otherwise, I join them with a single space because they are presumed to be two parts of a single paragraph split across two pages.

It should be obvious now where we’re going with this. We’ll have code that does something like this:

# If the paragraph does not end with final punctuation, accumulate it
if not is_sentence_end(p_str):
    combined_paragraph = combine_paragraphs(combined_paragraph, p_str)
    combined_chars += p_str_chars
    continue

If a Docling text item doesn’t end like a regular sentence, then we’ll try to combine it to the next one. So, something like this in a PDF:

Now becomes:

“Hence I suggested that testability or refutability or falsifiability should be accepted as a criterion of the scientific character of theoretical systems; that is to say, as a criterion of demarcation between empirical science on the one hand and pure mathematics, logic, metaphysics, and pseudo-science on the other.”

The page footer (page number) and page header are removed in between and the dashes are removed.

Conclusion

We’re finally ready to pull it all together – in the next blog post. We’ll finally get a working version of our DoclingParser.

IBM’s Docling for Superior Text Loading from PDFs

Tue, 04 Mar 2025 09:00:00 -0700

Now that we have NLTK installed and some code written to remove hyphens and dashes from PDF text, it’s time to put this to work for a first-class PDF loader. In this post, I’ll go over the basics of using Docling to get an improved text load from a PDF.

I previously covered how to use IBM’s Docling to turn a PDF into Markdown. (See this post here. ) At the time, I had the idea of converting a PDF to Markdown with Docling, translating it to HTML, and then running it through my existing HTML parser (as covered in a past post.) But when I actually tried it, the results were less than impressive.

As discussed in this post, PDFs are much more challenging than EPUBs because EPUBs are essentially HTML. They contain convenient tags for titles and section headers, the text is always unhyphenated, and it's easy to determine when a paragraph ends—even if it continues onto the next page.

PDFs, on the other hand, lack these advantages because they are designed primarily for visual layout, not for extracting readable text and storing it in an AI document database.

The challenge is that PDFs are far more common than EPUBs, so we need a reliable way to convert them into high-quality text for our document store.

IBM’s Docling to the Rescue

I’ve struggled to find an open-source PDF parser that I really like. In this post, I tested several popular open-source tools for extracting text from PDFs and found them all lacking. Then I discovered Docling—and at first, I didn’t like it either.

But after digging deeper, I was pretty impressed. Docling uses machine learning to tag text with various labels, allowing it to distinguish between the main body text and elements like section headers, page headers, or footers. Since it relies on machine learning, it’s not perfect—more on that later.

Still, this is the closest I’ve seen to treating a PDF like an EPUB, so I’ve decided to build a Docling parsing class around this capability and refine the results to my satisfaction.

The one thing Docling didn’t do for me was remove those pesky hyphens and dashes that break words at the end of a line in a PDF. This made the extracted text look terrible.

It also struggled with identifying paragraph breaks, performing only so-so in that regard. And when it came to detecting whether a paragraph continued onto the next page—especially with a page header or footnotes in between—it failed completely (as in, not at all).

Still, Docling is a huge step up compared to most PDF readers, so it will now serve as the foundation for loading PDFs into the Book Search Archive—Mindfire’s toy app for testing our growing open-source stack (and the basis for most of these blog posts). You can find the code as it was at the time of this blog post inside the Book Search Archive `.

The key to using Docling effectively is not to rely on its PDF-to-Markdown conversion (as most beginner tutorials suggest). Instead, it’s better to iterate over the text using Docling’s built-in objects and leverage its labels to determine how to handle the extracted content.

The Power of Docling: A Code Example

Let’s look first at some of the code in the customhaystackcomponents.py file. Look at the new component called DoclingParserComponent. This component is pretty simple. It simply takes a list of PDFs and converts them into a list of documents (as ByteStreams) and a list of meta data and then returns those lists onto the next step in the Haystack pipeline:

@component.output_types(sources=List[ByteStream], meta=List[Dict[str, str]])
def run(self, sources: List[DoclingDocument], meta: List[Dict[str, str]]) -> Dict[str, Any]:
    docs_list: List[ByteStream] = []
    meta_list: List[Dict[str, str]] = []

    for i, doc in enumerate(sources):
        meta_data: Dict[str, str] = meta[i]
        parser: DoclingParser
        start_page: Optional[int] = None
        end_page: Optional[int] = None
        if doc.name in self._valid_pages:
            start_page, end_page = self._valid_pages[doc.name]
        parser = DoclingParser(doc, meta_data,
                               min_paragraph_size=self._min_paragraph_size,
                               start_page=start_page,
                               end_page=end_page,
                               double_notes=True)
        # Start here
        temp_docs: List[ByteStream]
        temp_meta: List[Dict[str, str]]
        temp_docs, temp_meta = parser.run()
        # item_id: str = meta_data.get("item_id", "")
        book_title: str = meta_data.get("book_title", "")
        # Unlike EPUB we don't have sections or chapters. So we don't need a total length.
        # TODO: Add a way to skip pages instead.

        self._print_verbose(f"Book: {book_title};")
        docs_list.extend(temp_docs)
        meta_list.extend(temp_meta)

    return {"sources": docs_list, "meta": meta_list}

All the real work is done by the parser:

temp_docs, temp_meta = parser.run()

So, let’s dig into the DoclingParser component as found in the docling_parser.py file. I’ll explain this file fully in future posts, but for now let’s keep it simple. You basically create a parser for a specific PDF and then ‘run’ to get the result:

        parser = DoclingParser(doc, meta_data,
                               min_paragraph_size=self._min_paragraph_size,
                               start_page=start_page,
                               end_page=end_page,
                               double_notes=True)

        temp_docs, temp_meta = parser.run()

This may not have been the best approach for me. I started down one path, thinking this would be a Haystack component on its own, but then changed my mind halfway through. I’ll clean it up in the future, but for now, it works just fine.

With the DoclingParser, you can specify various parameters, like whether to enable ‘double_notes’ (i.e., allowing a double minimum size for footnotes so they don’t dominate the semantic search we’ll do later). You can also pass starting and ending pages, which I’ll specify via a CSV file, just like we did for sections in our EPUB parser. This setup allows you to exclude elements like the Introduction, Table of Contents, and Index—things that aren’t helpful for our document semantic search. But all the real work is in the DoclingParser class itself.

The DoclingParser Class: Using Docling Labels to Improve Text

The key work loop is as follows:

Convert the DoclingDocument passed (in the constructor) to a list of various items. (See also full documentation found here.)
Enumerate over that list and process it.
Collect meta data (e.g. the page number in this example) to save off.)

texts = self.getprocessed_texts()

for i, text in enumerate(texts): nexttext = getnexttext(texts, i) pageno = getcurrentpage(text, combinedparagraph, pageno)

But what does getprocessedtexts do? Honestly, not much:

def _get_processed_texts(self) -> List:
    # Before we begin, we need to find all footnotes and move them to the end of the texts list
    # This is because footnotes are often interspersed with the text, and we want to process them all at once
    # Split texts into regular content and notes (footnotes + bottom notes)
    regular = [t for t in self._doc.texts if not (is_footnote(t) or is_bottom_note(t))]
    notes = [t for t in self._doc.texts if is_footnote(t) or is_bottom_note(t)]
    return regular + notes

All I’m doing here is taking the ‘text’ attributer of the DoclingDocument passed in the constructor (stored in self._doc) and creating two lists. One (‘regular’) is a list of all texts in the DoclingDocument that aren’t footnotes or bottom notes. One (‘notes’) is just a list of footnotes and bottom notes. I then move the footnotes and bottom notes to the end of the list of text items.

One thing to note here (though it isn’t obvious from the annotations – I need to fix that) is that ‘texts’ is a list that can be a number of different classes. For example:

Union[SectionHeaderItem, ListItem, TextItem])

A TextItem is just regular text in the body of the document. A ListItem is text in a bullet point style list. A SectionHeaderItem is a section header. All of these are kinds of DocItems. There are others, but these are the ones I’m currently playing with. In addition, all these items have a ‘label’ on them that contains a text label like one of these:

"section_header"
"page_footer"
"page_header"
"footnote"
"list_item"
"formula"
Etc.

What I next did is I wrote a number of methods to determine (based on the label attribute) which type of text this is:

def is_section_header(text: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    if text is None:
        return False
    return text.label == "section_header"


def is_page_footer(text: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    return text.label == "page_footer"


def is_page_header(text: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    return text.label == "page_header"


def is_footnote(text: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    return text.label == "footnote"


def is_page_not_text(text: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    return text.label not in ["text", "list_item", "formula"]


def is_page_text(text: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    return not is_page_not_text(text)

def is_text_item(item: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    return not (is_section_header(item)
                or is_page_footer(item)
                or is_page_header(item))

You should get the idea of where I’m going with this. It’s now very similar to our HTML parser from the EPUB posts. I’ll explain how I fully utilize these features in a future post. For now, the key takeaway is that Docling provides labels that help you determine what type of text you’re working with. I can even detect things like tables. This is where the real power of Docling lies.

One problem, though: it isn’t perfect. Page headers can end up as section headers. Paragraphs may break in the middle or across pages, sometimes even with footnotes in between. Occasionally, a paragraph might be inserted right in the middle of another one. Or it may think footnotes are regular text.

Because of this, I had to (for my PDFs) create some custom code to find, say, bottom notes that Docling thought were regular TextItems:

def is_bottom_note(text: Union[SectionHeaderItem, ListItem, TextItem]) -> bool:
    if text is None or not is_page_text(text):
        return False
    # Check for · at the beginning of the line. This is often how OCR represents footnote number.
    if text.text.startswith("·") and not text.text.startswith("· "):
        return True
    return bool(re.match(r"^\d+\S.*", text.text))

Here, you can see that I’m taking whatever text item is passed and—admittedly somewhat simplistically—I’m assuming that if it starts with a number or a ‘dot’ (because Docling sometimes OCRs footnote subscripts and interprets them as dots), then it’s likely a footnote. This is obviously very specific to the PDFs I’m working with and isn’t a general rule, but you get the idea.

Conclusions

This should give you the basic idea of the power—and limitations—of using Docling to parse your PDFs. We still have quite a few issues to address, and I’ll tackle those in the next post. But for now, we have a solid way to iterate over the text in the PDF and determine if it's text we want to skip (like page headers or footers with page numbers), if it’s text we want to capture into our document store (such as TextItem or ListItem), or if it’s text we want to capture as metadata (like SectionHeaderItem). We’re on our way to replicating our success with EPUBs, but this time with PDFs.

Also, don't forget that Mindfire TECH is here to not just provide these free articles but also to offer our services to help your business apply these AI principles and concepts. If you would like to learn more about how we can help you get your TECH moving with AI, please do reach out for a free consultation or discussion via our contact-us page!

Using NLTK to Improve RAG (Retrieval Augmented Generation) Text Quality

Thu, 27 Feb 2025 09:00:00 -0700

We’ve been working with Docling (and also PyMuPDF4LLM) to read and parse text from a PDF document. But PDFs present a challenge because, unlike EPUBs, they are designed for pure visual layout, not as a source of readable text. That means what looks great on a screen in a PDF might turn into a mess when you extract the text.

For example, consider this paragraph in a PDF:

This paragraph looks just fine when you read it in a PDF. But note the highlighted hyphens at the end of the line—your eyes are trained not to even notice them. However, when Docling processes this text, it’s going to look something like this:

It is not surprising to find such views in Bishop Berkeley. Indeed, Berkeley made it clear that he published them largely in the hope of defending religion against the onslaught of science and of 'free- thinking'; against the claim that reason, unaided by divine revela- tion, can discover a world behind the world of appearance. But it is a little surprising to find support for these instrumentalist views in the camp of the admirers of science.

I doubt it affects a semantic search too much (though it might a little), but it’s not the most pleasant to read. It would be nice if we could fix problems like this.

In addition, Docling sometimes inserts odd spaces in the wrong places or introduces other artifacts due to PDFs not really being meant to serve as a source of text the way we’re trying to use them. PDFs are, first and foremost, designed to be read on a screen—not fed into an AI for processing by a Large Language Model (LLM).

How to Remove Unnecessary Hyphens

How might we go about removing unnecessary hyphens? This isn’t a simple problem. You might think you could just remove all hyphens, but in some cases, hyphens are necessary—for example, in compound words or to set off part of a sentence for emphasis. There’s no perfect set of rules that removes only the hyphens you don’t want while keeping the ones you do.

You might consider doing it based on whether there’s a space or not. But hyphens used to break up a sentence often have no spaces, while hyphens that split a word at the end of a line often do. Unfortunately, this isn’t consistent either way, so that approach won’t work.

What we really need is to determine whether, once spaces are removed, the hyphen splits a single word or if there are full words on both sides of it.

For example, revela-tion is clearly part of a single word—revelation—because revela and tion aren’t words on their own.

But how would we accomplish that?

Installing NLTK

Luckily there is an existing library called NLTK (Natural Language Toolkit) that can help us out here.

First, we need to install it into our Python environment by doing the following:

pip install nltk

I’m here assuming you already have Numpy installed – which you do if you did the environment setup already for the “Book Search Archive”.

You can find the version of Book Search Archive at the time of this post (which includes the code below) at this link.

Using NLTK

To make this work we’ll first need to download some libraries for use in NLTK:

nltk.download('words')
nltk.download('wordnet')
nltk.download('omw-1.4')

Then you’ll need to create a Python set of words:

words_list: set = set(nltk.corpus.words.words())

And then setup the lemmatizer and stemmer:

lemmatizer = WordNetLemmatizer()
stemmer = PorterStemmer()

What is a lemmatizer and stemmer? A lemmatizer returns the base of a word. So:

print("Lemmatization:")
print("caring →", lemmatizer.lemmatize(word1, pos="v"))   # caring → care
print("geese →", lemmatizer.lemmatize(word2))            # geese → goose
print("troubled →", lemmatizer.lemmatize(word3, pos="a")) # troubled → troubled (lemma is the same)

And:

print("\nStemming:")
print("caring →", stemmer.stem(word1))  # caring → care
print("geese →", stemmer.stem(word2))   # geese → gees (incorrect root)
print("troubled →", stemmer.stem(word3)) # troubled → troubl

So:

In other words:

Lemmatization considers the word’s meaning and grammar (e.g., geese → goose).
Stemming blindly chops off word endings (e.g., geese → gees, troubled → troubl).

Unfortunately, even using the lemmatizer and stemmer, I still found that it just didn’t always find the best places to remove hyphens. So I also built some ‘custom lemmas’ of my own for common problems:

    suffixes = {
        "ability": "able",  # testability -> testable
        "ibility": "ible",  # possibility -> possible
        "iness": "y",         # happiness -> happy
        "ity": "e",          # creativity -> create
        "tion": "e",         # creation -> create
       "able": "",         # testable -> test
        "ible": "",         # possible -> poss
        "ing": "",          # running -> run
        "ed": "",           # tested -> test
        "s": ""             # tests -> test
    }

Finding a Valid Word Using NLTK

With this all in place, here is my isvalidword() function that I created to check if the two sides of a hyphen contained valid words or not:

def is_valid_word(word):
    """Check if a word is valid by comparing it directly and via stemming."""
    stem = stemmer.stem(word)
    if (word.lower() in words_list
            or word in words_list):
        return True
    elif (stem in words_list
          or stem.lower() in words_list):
        return True
    # Check all lemmatizations of the word
    options = ['n', 'v', 'a', 'r', 's']
    for option in options:
        lemma = lemmatizer.lemmatize(word, pos=option)
        if lemma in words_list:
            return True
    # Check for custom lemmatizations
    suffixes = {
        "ability": "able",  # testability -> testable
        "ibility": "ible",  # possibility -> possible
        "iness": "y",         # happiness -> happy
        "ity": "e",          # creativity -> create
        "tion": "e",         # creation -> create
        "able": "",         # testable -> test
        "ible": "",         # possible -> poss
        "ing": "",          # running -> run
        "ed": "",           # tested -> test
        "s": ""             # tests -> test
    }
    for suffix, replacement in suffixes.items():
        if word.endswith(suffix):
            if suffix != 's':
                pass
            stripped_word = word[: -len(suffix)] + replacement
            if is_valid_word(stripped_word):
                return stripped_word

    return False

The code does these checks:

Is the text passed found in the list of words we previously downloaded?
Is the stem of the text passed in the word list?
Is the lemma of the text passed in the word list?

Note that there are 4 kinds of ‘lemmatizing’ called n, v, a, r, and s. We won’t worry, for now, what those are. We’re just going to check every single kind.

Finally, we try replacing various suffixes from my custom list with alternatives more likely to properly lemmatize and then try again.

If any of these find it to be a word, we count the text passed as a word.

I’ve found in practice this function works pretty well. It probably has some room for improvement.

Now we’ve got what we need to be able to check to see if the hyphen is between two actual words or if it is separating a single word. (Obviously there is more logic required to use this function.

Pulling it All Together: The “De-Hyphenator”

Now let’s pull it all together and write a function that takes text for a paragraph and rewrites it to remove unwanted hyphens:

def combine_hyphenated_words(p_str):
    # This regular expression looks for cases where a dash separates two parts of a word
    # The idea is to combine the two parts and check if they form a valid word.
    def replace_dash(match):
        word1, word2 = match.group(1), match.group(2)
        combined = word1.strip() + word2.strip()
        first_word = p_str.strip().split(' ')[0]

        # does word 1 contain a space after the hyphen?
        if word2.startswith(" ") and is_valid_word(combined):
            # When there is a space after the hyphen, it is likely that the hyphen is separating two parts
            # of a single word
            return combined
        # else check for each part individually being a word. If so, this is probably a compound word
        elif is_valid_word(word1.strip()) and is_valid_word(word2.strip()):
            return word1.strip() + '-' + word2.strip()
        # else if the combined word is a valid word, then we probably had one word broken in two
        elif is_valid_word(combined):
            return combined  # Combine the parts if they form a valid word
        # if the combined word starts with a capital letter, then it is likely a proper noun. Combine the parts.
        elif combined[0].isupper() and not word2.strip()[0].isupper() and not is_valid_word(word2.strip()):
            return combined

        # Default - assume the hyphen is separating two words
        return word1.strip() + '-' + word2.strip()

    # Replace soft hyphen characters (¬) with a regular dash
    p_str = p_str.replace("¬", "-")
    # p_str = p_str.replace("- ", "-")

    # Look for dashes separating word parts (no spaces involved)
    p_str = re.sub(r'(\w+)-(\s?\w+)', replace_dash, p_str)

    return p_str

You can see that I have a sub-function called replace_dash() that does most of the work. The main function simply tries to place double hyphens with hyphens:

p_str = p_str.replace("", "-")

That may look like replacing a dash with a dash, but that first dash is really a double dash.

Next, we use re.sub (a regular expression substitute function) to find all strings of letters separated by dashes and run the replace dash function on it:

    p_str = re.sub(r'(\w+)-(\s?\w+)', replace_dash, p_str)
    return p_str

The replace_dash() function does the real work. It does the following:

Set word1 to be the text before the dash and word2 to be the text after the dash
Set the ‘combined’ variable to be a concatenation of the two.
Check if word2 starts with a space (meaning word1 was text-dash-space meaning likely it was found at the end of a text line) and the combined word is valid. If so, return the combined word.
Check if word1 and word2 are each individually valid. If so, return them with a dash in between.
Check if the combined word is valid. If so, return combined word. (Similar to #3 but lower priority because there is no sign that it happened at the end of the text line in the PDF)
If the combined word starts with an upper case and word2 does not and word2 is not valid on its own, return the combined word. This is likely a proper noun split up, so we’re recombining.
Finally, the default is to just return it back as word-dash-word.

Many of these rules probably sound quite similar. But the order of operations matters here. This function does not always work right, but works pretty well most of the time. If you have ideas how to improve, let me know.

Conclusions

NLTK is another tool in the box for AI develop. LLMs deal with language and NLTK is there to assist you. We only used it for a very simple purpose – check for if a word is valid or not to remove unneeded hyphens – but NLTK has a lot more to offer.

Docling for PDF to Markdown Conversion

Tue, 18 Feb 2025 12:00:00 -0700

Back in this post, I went over several ways to try to convert a PDF into document fragments for our Book Search Archive (the Mindfire toy app for our open-sourced AI stack.) None of them worked great. One reader of these posts suggested I try out Docling. (Github Repo for Docling. Documentation for Docling.)

Docling is IBM’s open-source library for reading popular document formats – including PDF – and exporting it to Markdown. It is similar to PyMuPDF4LLM in that he attempts to remove headers and footers and other extraneous information from a PDF so that you are only creating embeddings for the ‘good parts’ of the document.

If you’ll recall, on my first attempt to use PyMuPDF4LLM it didn’t work well for me and kept repeating words. Will Docling work better? Unfortunately, the answer turned out to be – nope! In fact, Docling was considerably worse than PyMyPDF4LLM in terms of results. Most of the book I tried just disappeared into a giant list of repeating words.

This did make me wonder if maybe the problem was this particular PDF. Perhaps, by dumb luck, I happened to grab a PDF that was really awful. In a future post, I will try other PDFs using both PyMuPDF4LLM and Docling and report back the results.

But for this post, let’s just go over how to install Docling plus the code I added to the Book Search Archive to make it a new PDF option. You can find my code for the Book Search Archive at the time of this post here.

Installing Docling

Installing Docling is very easy. Just run this command:

pip install docling

One word of warning here: Docling needs to install a large number of libraries to work. Far more than I recall getting installed for PyMuPDF4LLM. So, I wouldn’t recommend doing what I did and installing both Docling and PyMuPDF4LLM. Pick one that works best for you and go with it.

Integrating Docling into the Book Search Archive

That being said, my intentions for the Book Search Archive are to test out various open-source software libraries, so it made sense for me to try out both. Further, I like the idea of having both available in case one works better on some PDFs and one on other PDFs. If in testing I find this to be the case I’ll write a way to specify which PDFs use which library.

So here is how I integrated Docling into my code. First, I wrote a new custom component that utilizes Docling to convert PDFs to mark down (custom_haystack_components.py):

@component
class DoclingToMarkdown:
    def __init__(self, min_page_size: int = 1000):
        self._min_page_size = min_page_size
        self._converter = DocumentConverter()

    @component.output_types(sources=List[ByteStream])
    def run(self, sources: List[str]) -> Dict[str, List[ByteStream]]:
        markdown_docs: List[ByteStream] = []
        for source in sources:
            markdown_doc: str = self._converter.convert(source).document.export_to_markdown()
            byte_stream: ByteStream = ByteStream(markdown_doc.encode('utf-8'))
            markdown_docs.append(byte_stream)
        return {"sources": markdown_docs}

We need to instantiated a Docling DocumentConverter in the __init__ method and save it to an instance variable and then use it to create a markdown document:

markdown_doc: str = self._converter.convert(source).document.export_to_markdown()

Now we need to use this component to build our document conversation pipeline (document_processor.py). First let’s add a new PDF conversion strategy to our enum:

class PDFReadingStrategy(Enum):
    PyPDFToDocument = 1
    PDFReader = 2
    PyMuPdf4LLM = 3
    PyMuPDFReader = 4
    Docling = 5

Then we need to add some code to the _doc_converter_pipeline method. First add two new components, conditional on the Docling strategy:

elif pdf_reading_strategy == PDFReadingStrategy.Docling:
    doc_convert_pipe.add_component("pdf_loader", DoclingToMarkdown())
    doc_convert_pipe.add_component("markdown_converter", MarkdownToDocument())

And then connect the components:

elif pdf_reading_strategy == PDFReadingStrategy.Docling:
    doc_convert_pipe.connect("pdf_loader.sources", "markdown_converter.sources")
    doc_convert_pipe.connect("markdown_converter.documents", "epub_pdf_merger.pdf_docs")

We connect the pdf_loader (i.e. the Docling custom component) to a Haystack built-in MarkdownToDocument component. The rest of the pipeline can stay the same. Here is the updated pipeline:

Note that I integrated Docling directly rather than using the built-in Haystack Docling integration via their DoclingConverter component. (See GitRepo here.) I’ll try that out in some future post. And that’s it! We now have Docling added to our open-source stack! Now we can try out comparisons between Docling and PyMuPDF4LLM or even use whichever one works best for a particularly document.

Llama.cpp for Large Language Models

Wed, 12 Feb 2025 11:00:00 -0700

In a previous post, we tried Ollama software to run our Large Language Models (LLM). Ollama seemed to be an improvement overloading the model using the Haystack Hugging Face component – probably mostly because it pre-loads and caches the models when the service starts up.

Llama.cpp (pronounced Llama C++) is another way to run LLMs similar to Ollama. However, it is written from the ground up in C++ for efficient inference of LLMs. And Haystack has a built-in integration component: The Haystack LlamaCppGenerator component.

The Advantages of Llama.cpp

Llama.cpp leverages the efficient quantized GGUF format. While this will reduce memory requirements and accelerating inference, it means you have to actually download the GGUF file directly to run the model. Yes, that’s right. Unlike the Ollama or Hugging Face interface where you can just pass a model name, you must actually download the GGUF file first yourself.

To try to abstract this a way a bit, I’ve created a model wrapper (as I did with all the other model interfaces we’ve tried out) that includes an automatic download of the model if it isn’t already downloaded. The Haystack Llama.cpp integration documentation suggests how to do this. I just integrated it into my code and simplified it a bit. Here is the wrapper I wrote that my RAG pipeline uses:

class LlamaCppModel(GeneratorModel):
    def __init__(self,
                 model_link: str = 'https://huggingface.co/TheBloke/openchat-3.5-1210-GGUF/resolve/main/openchat-3.5-1210.Q3_K_S.gguf',  # noqa: E501
                 context_length: int = 2048,
                 max_tokens: int = 512,
                 temperature: float = 0.6,
                 verbose: bool = True) -> None:

        super().__init__(verbose=verbose)
        self._warmed_up: bool = False
        self._model_link = model_link
        # Take name of the model from the link. Everything after the last /
        self._model_name = model_link.split("/")[-1]
        self._context_length = context_length
        self._max_tokens = max_tokens
        self._temperature = temperature

        if self._verbose:
            print("Warming up LlamaCPP Large Language Model: " + self._model_name)

        # Check if model is already downloaded and download if necessary
        self._download_model()
        self._model: LlamaCppGenerator = LlamaCppGenerator(
            model=self._model_name,
            n_ctx=self._context_length,
            n_batch=512,
            model_kwargs={"n_gpu_layers": -1},
            generation_kwargs={"max_tokens": self._max_tokens, "temperature": self._temperature},
        )

    def generate(self, prompt: str) -> str:
        return self._model.run(prompt)

    @property
    def context_length(self) -> Optional[int]:
        return self._context_length

    def warm_up(self) -> None:
        if not self._warmed_up:
            self._model.warm_up()
            self._warmed_up = True

    def _download_model(self):
        # Checks if the file already exists before downloading
        if not os.path.isfile(self._model_name):
            urllib.request.urlretrieve(self._model_link, self._model_name)
            print("Model file downloaded successfully: " + self._model_name)
        else:
            print("Model file already exists: " + self._model_name)

You have to pass to the model wrapper the full URL for the GGUF file you’re interested in using like this:

model: gen.GeneratorModel = gen.LlamaCppModel(model_link="https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/resolve/main/zephyr-7b-beta.Q4_K_M.gguf")

You’ll also need to install the Haystack component for the Llama.cpp integration:

pip install llama-cpp-haystack

This will also automatically install llamacpppython which is a python wrapper for llama.cpp (which is of course written in C++ for speed. Note: I remember back when C++ was considered the slow language. 😊) You can find the GitHub repo for llamacpppythong here.

Picking a Model

You might be wondering how to find the GGUF model file to use. Not all Hugging Face Models have one. In fact, most official files don’t. But the community often quantizes official models and then puts them up on the Hugging Face website and ecosystem. “TheBloke” is famous for this, so let’s try out one of his and I’ll walk you through how to find the full path to the file within the Hugging Face website.

The Haystack integration documentation suggests this model:

https://huggingface.co/TheBloke/openchat-3.5-1210-GGUF

But I wanted to try a quantized version of the zephyr-7B-beta model because that is the model we use via the Hugging Face API (as discussed back in this post). It can be found here:

https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF

What is Quantization?

Perhaps you are wondering what quantization is? When an LLM is initially trained, floating point numbers are used to represent the weights. This is usually represented via 16-bits, or possibly 32-bits. To quantize the model, we compress the values of the weights down to a smaller number of bits. The values stay the same, but some accuracy of the floating-point values is lost.

The result is the same model with the same weights – with a bit of loss of accuracy in the weights – but compressed down to a much smaller size. This means the model takes less memory and will generate responses (i.e. inference) faster.

How to Find the GGUF Model File

Let’s start by navigating to the link of the model we are going to use:

Scroll down a bit and there is a list and description of your options:

I’m going to target the 4-bit quantized model. That means we’re squeezing 16-bits down to 4-bits! But notice the comment next to the zephyr-7b-beta.Q4KM.gguf model: “medium, balanced quality – recommended”. Sounds promising. Click the link to that file:

https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/blob/main/zephyr-7b-beta.Q4KM.gguf

And then click here and it will copy a download link into your clipboard:

For me that was:

https://huggingface.co/TheBloke/zephyr-7B-beta-GGUF/resolve/main/zephyr-7b-beta.Q4KM.gguf

That is the link you want to pass to my model wrapper. The wrapper will then download the gguf file. If it was previously downloaded it will use the downloaded file.

You can find the updated code as of writing this blog here.

The Disadvantages of Llama.cpp

So, Llama.cpp is faster than other methods of running an LLM, but it has some definite disadvantages – beyond having to find your own GGUF model file. While Llama.cpp does support streaming (the LangChain integration utilizes streaming using Llama.cpp) Haystack’s Llama.cpp integration does not support steaming yet.

Conclusions

Llama.cpp is a great way to implement a production LLM locally so that you don’t need to let your company’s private data visible to OpenAI or Microsoft. It’s support for quantized models and C++ implementation makes it a superior choice. However, it requires a bit more work to get it up and running.

Links:

Llama.cpp Tutorial: https://www.datacamp.com/tutorial/llama-cpp-tutorial
Llama.cpp official documents: https://llama-cpp-python.readthedocs.io/en/latest/api-reference/#llama_cpp.Llama
Llama.ccp Github Repo: https://github.com/ggerganov/llama.cpp
A Llama.cpp Python Interface Repo: https://github.com/abetlen/llama-cpp-python
Llama.cpp Haystack Integration: https://haystack.deepset.ai/integrations/llama_cpp
My feature request for streaming callbacks: https://github.com/deepset-ai/haystack/issues/8682

Explaining DeepSeek R1 (and How to Use It)

Tue, 04 Feb 2025 09:00:00 -0700

If you follow AI at all, you've probably been hearing across nearly all media about the sudden and unexpected rise of a new Large Language Model (LLM) from DeepSeek—based in China, no less!—that’s making waves and worrying American companies. So, what’s going on, and how can you take advantage of it?

Who is DeepSeek?

DeepSeek is a China-based company that has actually been in the LLM space for a while. Prior to R1 they released DeepSeek-V3-Base as their premier LLMs. But they were really just one model amongst many until they recently made a big splash with the highly visible release of their R1 model.

What is the DeepSeek R1 Model?

DeepSeek’s R1 came close to or even exceeded the benchmarks of all the latest and greatest LLMs out there including Open.Ais O1 and Anthropic’s Claude.

“R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks.” (p. 1 of “DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning”. All quotes from this paper.)

The above graph is from DeepSeek’s excellent paper the published on their R1 model.

Now, I can hear you objecting: “Sure, that’s impressive and all, but so what? It’s barely better than the other models. And how reliable are LLM benchmarks anyway? Do they even mean anything?” And you’re right. Benchmarks don’t mean much, and the improvement isn’t that significant. So why is this considered such a major disruption, sending American companies into a panic?

Why Does R1 Scare Everyone?

DeepSeek claims they trained R1 on a budget of $5-$6 million. There are some claims this is an exaggeration. However, even if it cost some multiple of this, it is still less than the hundreds of millions or possibly billion or so often spend on training LLMs. Rumor has it that the revelation that you can train LLMs for far cheaper and less compute caused the sudden drop in the price of Nvidia’s stock. But this doesn’t really make sense because making LLMs cheaper to use will only increase demand for them, leading to increased demand for GPUs.

On top of the lower training costs, R1 has 671 billion parameters which is smaller than, say, GPT-4’s rumored 1.8 trillion parameters. But its architecture only activates 37 billion at a time which makes it pretty computationally efficient.

How R1 Was Created

The DeepSeek-R1 project teaches AI models to "think better" using a special kind of training where the model learns on its own through trial and error—no textbooks or examples needed (p. 2, 5). Instead of giving it answers, the team set up rules that make the model show its work, like solving math problems step-by-step before giving a final answer (p. 6). Over time, the model (called DeepSeek-R1-Zero) started doing clever things naturally, like double-checking its own answers, trying different approaches, or even having "aha moments" where it suddenly fixes its mistakes after rethinking (p. 8). It’s like the model figured out how to think, not just what to think.

The catch? DeepSeek-R1-Zero’s answers can be messy—sometimes mixing languages or missing clear formatting, making them hard to read (p. 10). But overall, it shows how letting AI learn by trying (and failing) can unlock surprising problem-solving skills. This led to training a second model with some cold start data to help develop the model’s thinking in a way comprehensible to humans. That became the DeepSeek-R1 Model. (Which is distinct from the original DeepSeek-R1-Zero model first developed using reinforcement learning.)

But the end results are impressive: on tough math tests (AIME 2024), it jumped from scoring 15.6% to 71% correct—and even 86.7% when voting on multiple answers, matching top AI models like OpenAI’s (p. 3). It’s also great at coding, beating 96% of humans in programming contests (p. 4).

Reinforcement Learning and LLM Development

Famously LLMs utilize human reinforcement learning to train the model to act in ways helpful (and safe) for humans. But R1 decided to use regular reinforcement learning to train the model to be excellent at what is known as ‘chain-of-thought’.

I will have to cover chain-of-thought in detail in a future post, but it’s a remarkably simple (but effective!) concept. You basically prompt the large language model to think step-by-step before giving an answer. The model will then naturally not jump straight to a (possibly wrong) answer but instead try to write some text about the problem and steps to solve it. This proved to be a powerful way to allow LLMs to score higher on reasoning benchmarks but at the expense of more tokens used to allow the LLM to ‘think’ about the problem.

Chain-of-thought will generally improve reasoning benchmark scores for models over 100B parameters. (Link). Open.AI incorporated chain-of-thought into their premium models and even fine-tuned their models to improve speed performance.

What R1 did was use rewards via Reinforcement Learning to reward good chain-of-thought reasoning in a certain format (for example to place the thinking inside tags.) As hinted above, this caused the model to discover ways to reason better even on smaller models.

If you are interested in learning more about Reinforcement Learning, I’ve created two videos on the subject that jump into the mathematics and how it works. Reinforcement Learning continues to amaze with it’s almost ‘general’ ability to learn. Learn Q-Learning (one of the best kinds of Reinforcement Learning) in this video. Then I show how to turn that into Deep Reinforcement Learning in this video.

Distilling R1

Once the R1 model was completed, DeepSeek took a curate sample of 800k created by R1 and used it to teach smaller, simpler AI models (like Qwen and Llama) using DeepSeek-R1’s "thinking patterns." Surprisingly, this worked better than training those smaller models directly, and some of these smaller models now outperform much bigger rivals (p. 3). All these models are free for anyone to use (p. 3).

You can find them here on Hugging Face because they are open-source.

The Deadly Combo

It was these three things together that made such an impact. A smaller Large Language model than its competitors performing as well as the state-of-the-art reasoning models but built for a fraction of the cost using a new method the ‘big boys’ didn’t even know existed. And then the buggers made it all available for free! The nerve!

It’s rumored that Meta started looking over their AI team – which each got paid as much as it took to train R1 – and started to wonder what was going on.

How To Get Started with R1

The easiest way to get started with R1 is their free interface found at chat.deepseek.com. Be sure to click the “DeepThink (R1)” button and you’ll immediately see the chain-of-thought reasoning in action.

I’ve been very impressed with the results. In fact, I asked it a difficult philosophical question that I don’t think even has an answer on the internet and R1 surprisingly came up with a pretty convincing answer I’ve never heard from a human being.

If you want to run R1 locally, another great way to get started it to download LM Studio. They have the various R1 models (including the distilled models and quantized versions of the models) available.

In future posts I’ll go over better ways to run R1 locally, such as Llama.ccp and Ollama.

Installing Ollama for Large Language Models (LLM) in Windows

Tue, 28 Jan 2025 09:00:00 -0700

Installing Ollama for Large Language Models (LLM) in Windows

I kept reading about Ollama for Large Language models (LLM), so I decided to download and try it out. Ollama is basically just software that lets you run an LLM. We’ve covered other software for running LLMs. I mentioned that the easiest way to do this is to use LM Studio. (See https://lmstudio.ai/) And of course we’ve used Haystack and Hugging Face all throughout my blog posts to run LLMs. They have their own software for that.

Ollama is a good alternative to LM Studios. It is easy to download and start using plus it has support in Haystack (see below) So it is somewhere between LM Studio in that an end user can run LLMs using it (though with a command line interface instead of a nice GUI) and it can also integrate with your Haystack or Hugging Face stack. Plus, being a command line, it has a lot more options available than LM Studio did.

That being said, I admit that I’m not overly impressed with Ollama. It doesn’t seem to ‘do much’ as it were that couldn’t just be done directly via Haystack or Hugging Face. Perhaps I’m missing the real point.

Loading Ollama

Be that as it may, let’s learn how to install Ollama and (in our next point) integrate it into our stack using Haystack. This post will cover installation on Windows (though it shouldn’t be much different for other operating systems.)

First navigate to the Ollama website at https://ollama.com/. You should see this:

Click the download button in the middle of the page and it will take you to this page: https://ollama.com/download. You will see this:

For me, I selected Windows, which seems to be the default. Or you can navigate here: https://ollama.com/download/windows

Now click the download button and you’ll download OllamaSetup.exe.

The instructions for Ollama are found on their github page. This includes install instructions similar (but more detailed) than this post.

When you are read, run the OllamaSetup.exe you downloaded and let it install. You should see this:

Click install and let it install.

It should run a small service, but if not, you can run Ollama directly by searching for Ollama app like this:

Once you run it, it will appear as a service like this:

Running Ollama in a Terminal

Now let’s run Windows Terminal. Presumably you already know how to do that but just in case, search for it like this:

From here you can start a model using the run command like this:

i.e. Ollama run gemma2 in this case.

Ollama will automatically download the model you choose, store it locally, and then run it and you’ll get a session like this:

FYI, that answer gemma 2 gave to my question was almost pure hallucination. Sigh.

Finding a Model

The Ollama website has a list of standard models found on this web page: https://ollama.com/search

However, Ollama also supports a community of models similar to Hugging Face ecosystem and you can search this set of custom models by searching using this search bar (on the home page!) instead of the official one:

There are a huge number of official and community models available. Let’s take a look at the Llama3.3 model page:

You can select the number of parameters you want your model to have (in this case, 70b) and then copy the Ollama command to run it:

You are now up and running with Ollama and local LLMs! Easy, right?

The C Drive Problem

One thing I rather dislike about Ollama is that it installs only to the C Drive and doesn’t let you specify where to download models to other than – of course! – the C drive. However, here are some helpful links that should get you around that problem. (Here and here.)

Integrating Ollama into Book Search Archive

I next went on to integrate Ollama into my open-sourced Book Search Archive to see how it performed compared to other models. You can find the codebase at the time of this blog post here.

First, you’ll need to install the needed integration with Haystack:

pip install ollama-haystack
I wrote a new wrapper just for Ollama models that looks like this:
class OllamaModel(StreamingGeneratorModel):
    def __init__(self,
                 model_name: str = 'gemma2',
                 url="http://localhost:11434",
                 temperature: float = 0.6,
                 streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
                 verbose: bool = True) -> None:

        super().__init__(verbose=verbose, streaming_callback=streaming_callback)

        if self._verbose:
            print("Warming up Ollama Large Language Model: " + model_name)

        self._model: OllamaGenerator = OllamaGenerator(
            model=model_name,
            url=url,
            streaming_callback=self._default_streaming_callback_func,
            generation_kwargs={
                "temperature": temperature,
                # "num_gpu": 1,  # Number of GPUs to use
                # "num_ctx": 2048,  # Reduce context window
                # "num_batch": 512,  # Reduce batch size
                # "mirostat": 0,  # Disable mirostat sampling
                # "seed": 42,  # Set a fixed seed for reproducibility
            },
        )

    def generate(self, prompt: str) -> str:
        return self._model.run(prompt)

Note that I now have a StreamingGeneratorModel abstract class to abstract away how streaming works. Previously only Hugging Face Models could stream. Now Hugging Face models and Ollama models can both stream. The key change is the call to Haystack’s OllamaGenerator component.

I created the model like this:

model: gen.GeneratorModel = gen.OllamaModel(model_name="gemma2")

Then pass that to the RAG pipeline. (I really need a better way to handle models than creating a custom wrapper for each one. I’ll add that to the TODO. It seemed like a good idea at the time. ☹)

Note that I’m using ‘gemma2’ instead of gemma-1.1-2b-it as I did for a local Hugging Face model. According to the Ollama model page, gemma 2 is a 9B parameters model! Yet it seems like it runs faster to me than the 2B parameters model I was using for Hugging Face. I’m not sure why this is. It might be mostly an illusion due to Ollama caching models whereas Hugging Face loads it each time. But even the generation – while not fast on my wimpy laptop – seems snappier than the 2B model I was previously using. So, color me impressed so far!

(Update: it occurred to me later that the default for Ollama Gemma2 model is only a 2k window of tokens whereas my Gemma 1 model defaulted to 8k for the context window. That explains why they run at about the same speed. ☹ But I’m still impressed by how quickly Ollama warms up. Whatever Ollama is doing seems to be effective. Probably because you actually warm it up when you start the service, and it is cached after that.)

Another nice feature is that you can pass a URL right into the OllamaGenerator Haystack class. That means you can use OllamaGenerator either locally or via an API and you can use the same generator. (Whereas with Hugging Face that was two different generators.) It’s not a huge deal, but makes it conceptually easier to treat local and API models the same like that. (Local just runs on localhost is all.)

And I did confirm that streaming works fine for OllamaGenerator. (Note that I pass self._default_streaming_callback_func instead of streaming_callback for a variety of complex reasons I explained in this post.)

Conclusions

Ollama is a great easy way to get started with local LLMs. It is more technical than LM Studios but also more flexible. I’m not sure if this is just my imagination, but it Ollama seems a lot faster than running a model locally (at least on my wimpy laptop) via Hugging Face. I was able to get okay results out of a 9B parameter Gemma 2 model.

However, in our next post I’ll cover You can find the official Haystack Ollama Integration Guide here.

Links - Matt Williams has an excellent basic and advanced courses on Ollama: https://www.youtube.com/watch?v=9KEUFe4KQAI&list=PLvsHpqLkpw0fIT-WbjY-xBRxTftjwiTLB

Reranking Documents Using Cross-Encoders for Retrieval Augmented Generation (RAG)

Tue, 21 Jan 2025 12:00:00 -0700

A while back I snuck a reranker into the Book Search Archive (Mindfire’s Open Source project to test out our open source AI stack). I forgot to do a write up for this blog, so I’m doing that now.

But what is a reranker? And why should you care?

To understand this, we have to harken back to the concept of cosine similarity and how we use it to do a semantic search on our document fragments. Recall that while cosine similarity works great for a semantic search (i.e. a search on meanings of words rather than a straight word search, i.e. a lexical search) but is quite slow because we have to do a cosine similarity computation on every single document and compare it to the query. To speed this up we use a Hierarchal Navigable Small World (HNSW) index. That works great and the results is very quick. But it only gives an approximate ‘nearest neighbor search’ result. That is to say, it works fast but as the expense that it may not find the best semantic search matches.

What the reranker does is it takes the top documents found by the HNSW index (as implemented in Postgres or Neo4j in this case) and then reranks them in memory. This may sound like a waste since the HNSW already pulled back the best hits. What use is it to rerank them?

Imagine you use the HNSW to pull back, say, the top 100 hits and then rerank those and take the top 5? The results would, ideally, include a solid hit that the HNSW network missed due to being only an approximate match. Plus, rerankers have all top 100 to work with so they can theoretically have more information to work with when coming up with the best results.

I got the idea to use a reranker while reviewing this white paper by Wang et all., 2024. (See also this excellent little article summarizing the results of Wang’s paper..) This paper goes over the best practices and best open-source models for a superior RAG pipeline. It hadn’t occurred to me to use a reranker prior to that point. And I admit I was skeptical that it would make much of a difference until I tried it and saw the positive results for myself.

Cross Encoding

A reranker utilizes a cross-encoder. Sbert.net describes a cross-encoder as follows:

Calculates a similarity score given pairs of texts.
Generally provides superior performance compared to a Sentence Transformer (a.k.a. bi-encoder) model.
Often slower than a Sentence Transformer model, as it requires computation for each pair rather than each text.
Due to the previous 2 characteristics, Cross Encoders are often used to re-rank the top-k results from a Sentence Transformer model.

A cross-encoder is specially trained for this task and so often gives better results than a regular cosine similarity via a sentence transformer model.

I tried out a reranker and it did successfully include an additional very good hit on a query that I’d done multiple times and the HNSW rankings had completely missed. Plus, it took the results the HNSW index came up with and put them into a new order that seemed to me to be better results. So, I’m sold that a good reranker using cross-encoding can really help your RAG pipeline.

Implementing a Reranker

The code for a reranker is pretty simple. It was so easy to implement that I forgot to make a blog post about it. 😊 I’m currently just using the TransformersSimilarityRanker component built into Haystack. This defaults to using the cross-encoder/ms-marco-MiniLM-L-6-v2 model. (Wang and company recommended this model instead. I should probably switch to that one but haven’t yet.)

I have a parameter for my RagPipeline class called use_reranker that, if set to True, will incorporate the reranker into the RAG pipeline.

First, in the __init__ method for the RagPipeline class, I added this code to warm up the reranker:

if self._use_reranker:
    # Warmup Reranker model
    ranker = TransformersSimilarityRanker(device=self._component_device, top_k=self._llm_top_k,
                                          score_threshold=0.20)
    ranker.warm_up()
    self._ranker = ranker

I set the score threshold to 0.20 which means it won’t return any documents with a similarity score below 20%. That’s a pretty low score, so you may want to raise the threshold to ensure only quality documents are used by the LLM.

Next, in the _create_rag_pipeline method I add the reranker into the actual RAG pipeline next to the doc_query_collector:

if self._use_reranker:
    # Reranker
    rag_pipeline.add_component("reranker", self._ranker)
    rag_pipeline.connect("doc_query_collector.documents", "reranker.documents")
    rag_pipeline.connect("doc_query_collector.query", "reranker.query")
    rag_pipeline.connect("doc_query_collector.llm_top_k", "reranker.top_k")
    # Stream the reranked documents
    rag_pipeline.add_component("reranker_streamer", DocumentStreamer(do_stream=self._can_stream()))
    rag_pipeline.connect("reranker.documents", "reranker_streamer.documents")

And then connect it to the prompt_builder:

if self._use_reranker:
    # Connect the reranker documents to the prompt builder
    rag_pipeline.connect("reranker_streamer.documents", "prompt_builder.documents")
else:
    # Connect the doc collector documents to the prompt builder
    rag_pipeline.connect("doc_query_collector.documents", "prompt_builder.documents")

And that is all there is to it. The new pipeline looks like this:

Conclusions

A reranker is a great addition to your RAG pipeline. It compensates for the short comings of an HNSW index not needing to do a cosine similarity across every document and possibly missing a good match and it tends to do a better job rating and ranking the documents.

Links

An excellent article explaining the advantages of using a Reranker

Using Neo4j Graph Database for Retrieval Augmented Generation (RAG)

Tue, 14 Jan 2025 12:00:00 -0700

In two previous posts (here and here) we talked about how to install Neo4j Graph Database for use with the Book Search Archive, our sample project using the Mindfire Technology open-source AI stack. In this post I’ll cover the code necessary to use Neo4j as a document store for Retrieval Augmented Generation (RAG).

You can find the version of the code at the time of this post here.

Adding Neo4j to Document Conversion Pipeline

Haystack’s Neo4j integration page discusses how to use the integration. Unlike the Pgvector integration with Haystack, the Neo4j integration was created and maintained by Neo4j and requires a separate install.

pip install sentence-transformers # required in order to run pipeline examples given below
pip install neo4j-haystack

You can find the Neo4j documentation on the Haystack integration here.

Next, we’ll add the Neo4j document store to the document_processor.py file. This new sub function (that I put inside of the _initialize_document_store method) can switch between the pgvector and Neo4j document stores:

class DocumentStoreType(Enum):
    Pgvector = 1
    Neo4j = 2
…
def init_doc_store(force_recreate: bool = False) -> Union[PgvectorDocumentStore, Neo4jDocumentStore]:
    if self._document_store_type == DocumentStoreType.Pgvector:
        connection_token: Secret = Secret.from_token(self._postgres_connection_str)
        doc_store: PgvectorDocumentStore = PgvectorDocumentStore(
            connection_string=connection_token,
            table_name=self._table_name,
            embedding_dimension=self.embed_dims,
            vector_function="cosine_similarity",
            recreate_table=self._recreate_table or force_recreate,
            search_strategy="hnsw",
            hnsw_recreate_index_if_exists=True,
            hnsw_index_name=self._table_name + "_hnsw_index",
            keyword_index_name=self._table_name + "_keyword_index",
        )
        return doc_store
    elif self._document_store_type == DocumentStoreType.Neo4j:
        # https://haystack.deepset.ai/integrations/neo4j-document-store
        doc_store: Neo4jDocumentStore = Neo4jDocumentStore(
            url=self._neo4j_url,
            username=self._db_user_name,
            password=self._db_password,
            database=self._db_name,
            embedding_dim=self.embed_dims,
            embedding_field="embedding",
            index="document-embeddings",  # The name of the Vector Index in Neo4j
            node_label="Document",  # Providing a label to Neo4j nodes which store Documents
            recreate_index=self._recreate_table or force_recreate,
        )
        return doc_store

Now just call this function like this and the rest of the code stays mostly the same:

document_store: Union[PgvectorDocumentStore, Neo4jDocumentStore]
document_store = init_doc_store()

There are various other small changes that I was forced to make to allow for the code to handle either type of document store, but other than that, not much else has changed.

Another thing to note is that the Neo4j document store (Neo4jDocumentStore) isn’t quite the same as the PostgreSQL document store (PgvectorDocumentStore). With the pgvector document store I needed to build a connection string out of the login information. With Neo4j I actually pass the login information into the document store. Also note that there is no flag to recreate the database in Neo4j and instead you just recreate the index, and it also recreates the database.

Querying the Graph Database

After running the new code and loading the Neo4j database, we can then go to the Neo4j browser and query the graph we created.

Let’s start with this query:

MATCH (n:Document) RETURN count(n) AS totalEntities;

For me, I’ve loaded 619 documents. Let’s create a query to view them as a graph:

Not much of a graph, huh? It’s just a bunch of nodes, each of which is a document fragment. So, at this point this is no different than just using the PostgreSQL/pgector database. We’ll go into detail how to take advantage of a graph database in future posts.

But what if you want to get the actual data back? This query might work for you:

MATCH (n:Document) 
RETURN {
    id: n.id,
    properties: [key IN keys(n) WHERE key <> 'embedding' | {key: key, value: n[key]}]
} AS document 
LIMIT 25

This cypher query actually returns all properties for a node. So, it’s just like querying a table in a regular SQL relational database.

Adding Neo4j to the RAG Pipeline

Now we’re ready to add the graph database to the RAG pipeline. We basically use the same trick. We create a function that will return the appropriate document store. It is nearly identical to the one we did in the document conversion pipeline. But this time we add it to rag_pipeline.py. After that we can add the nodes to the RAG pipeline to utilize the Neo4j database. This works the same way as before except with a new wrinkle: we can no longer use the PgvectorKeywordRetriever component. So lexical searches must be disabled if we’re using Neo4j.

if self._search_mode == SearchMode.LEXICAL or self._search_mode == SearchMode.HYBRID \
        and self._document_store_type == DocumentStoreType.Pgvector:
    lex_retriever: RetrieverWrapper = RetrieverWrapper(
        PgvectorKeywordRetriever(document_store=self._document_store, top_k=self._retriever_top_k))
    rag_pipeline.add_component("lex_retriever", lex_retriever)
    rag_pipeline.connect("query_input.query", "lex_retriever.query")
    rag_pipeline.connect("lex_retriever.documents", "doc_query_collector.lexical_documents")

if self._search_mode == SearchMode.SEMANTIC or self._search_mode == SearchMode.HYBRID:
    semantic_retriever: RetrieverWrapper
    if self._document_store_type == DocumentStoreType.Neo4j:
        semantic_retriever = RetrieverWrapper(
            Neo4jEmbeddingRetriever(document_store=self._document_store, top_k=self._retriever_top_k))
    else:
        semantic_retriever = RetrieverWrapper(
            PgvectorEmbeddingRetriever(document_store=self._document_store, top_k=self._retriever_top_k))
    rag_pipeline.add_component("semantic_retriever", semantic_retriever)
    rag_pipeline.connect("query_embedder.embedding", "semantic_retriever.query")
    rag_pipeline.connect("semantic_retriever.documents", "doc_query_collector.semantic_documents")

Notice how I check for each type of document store we want to use and adjust the pipeline either way. I also changed the RetrieverWrapper to accept either kind of document store.

Conclusions

At this point we’re not really utilizing the power of a graph database. But we have successfully changed our code to handle either a PostgreSQL or Neo4j database as the document store. So, our open-source AI stack now includes the power of graph databases should we need that ability. This is exciting!

If you're interested in learning more or are curious how you could use AI or ML technology in your current or future projects, be sure to reach out for a free consultation!

Installing Neo4j Graph Database Desktop Version

Tue, 07 Jan 2025 12:00:00 -0700

In a previous post, I covered how to install Neo4J. For the Mindfire stack, we’re aiming to stick with open-source software, and Neo4J Server Community Edition fits the bill. However, there’s also an Enterprise edition available. Interestingly, you can install and use the Enterprise edition for free—with some limitations.

Introducing Neo4J Desktop

Neo4j Desktop is free to download and includes a personal Neo4j Enterprise Edition Developer license. This allows you to access all the capabilities and features of the Neo4j Enterprise Edition, with the limitation that it can only be used by you and on a single machine. Unfortunately, this means the desktop version of Neo4j isn’t suitable for use in a production environment. However, it’s a much easier way to download, install, and get started with Neo4j, and it even lets you use the full Enterprise version for free. It also comes with some useful tools built-in to a convenient UI.

Recall where we downloaded Neo4J originally. If you scroll down a bit, you’ll find a download for Desktop version of Neo4J Graph Database. (You can also find a download page here. Find install instructions here.)

Click the download button and you’ll download and then run a file named something like this: Neo4j Desktop Setup 1.6.1.exe. Running this file will prompt you for where you want to store the database. Then you will be prompted to register your software:

Fill out the above form and you’ll get a registration key that you can fill in the software key here:

You may then be prompted for updates:

After the install is completed, you’ll see an interface like this:

Neo4J has some excellent documentation about the desktop version.

The Neo4J Desktop Interface

The Neo4J Desktop interface is a convenient way to work with Neo4J. Here you can see how I’m looking at my test databases via the UI:

The Desktop UI also has a number of built-in tools:

Select the top option, the Neo4J Browser, and you’ll get the Neo4J browser interface we introduced in the previous post:

And you’re ready to go!

Installing Neo4J Graph Database Self-Managed for Retrieval Augmented Generation (RAG)

Wed, 01 Jan 2025 12:00:00 -0700

In previous posts, we learned how to install PosgreSQL and then how to install pgvector (a vector search extension for PostgreSQL.) The Book Search Archive (our toy app to show off Mindfire’s growing low-cost open-source AI stack) currently utilizes a PostgreSQL database with vectors handled by pgvector to do our semantic searches using a HNSW index built-into pgvector.

The Woes of PostgreSQL and Graph Database Extensions

However, PostgreSQL is a standard relational database, and I really wanted to add graph database capabilities to our open-source stack. Lucky for me – or so I thought! – PostgreSQL has a graph database extension called Apache AGE. (GitHub repo here.) Apache AGE is the extension used for the “for pay” PuppyGraph database. So, I thought Apache AGE looked promising. I imagined in my mind having a single database that has regular relational tables, vector fields and indexes for semantic search, as well as graph database capabilities. It sounded incredible.

What’s that old saying about if it sounds too good?

It turns out that Apache AGE is not truly compatible with Windows (as noted here and here and here). I also had PostgreSQL 16 loaded and, at the time, Apache AGE did not support version 16. However, checking their website today, it looks like maybe they do now.

I’ll revisit Apache AGE as some future point. The temptation of having relational, graph, and vector databases all wrapped up in one stable open-source database is just too much for me. But for now, I’ve decided to try out a simpler solution for a Windows-based graph database: Neo4J.

Introducing Neo4J: An Amazing Open-Source Graph Database

Neo4J is a graph database that also has vector search support. Haystack also supports Neo4J with a built-in integration. So, at a minimum, I’d like the Mindfire open-source AI stack to provide support for Neo4J even if we decide later to go with Apache AGE as default.

What is a graph database you ask?

So, unlike a regular relational database where you have tables with rows, a graph database has nodes with named connections. Here is an example right off the Neo4J website:

As you can imagine, this will create a sprawling database of connections between ideas. Some kinds of data work much better organized as graphs rather than as tables. Plus, graph theory is basic to both computer science in general and AI in particular. So having graph database capabilities would be a real boon for many kinds of algorithms used in and out of Artificial Intelligence. (Find an introduction to graph theory here.)

Installing Neo4J

The good news is that installing Neo4J is quite a bit easier than trying to get PostgreSQL + pgvector to work. The bad news is that despite some extensive documentation on how to install and use Neo4J, they still screwed it up and made some mistakes. I’ll try to walk you through the correct way to avoid these mistakes.

Let’s start with the easy part. Browse to the Neo4J deployment center and find the version applicable for your operating system. For me (installing on Windows) it looked like this:

Be sure to select Community edition if you want the free open-sourced edition. For me, I selected Windows Executable and version 5.25.1. Strangely version 4.4.39 has a later release date than version 5.25.1. I’m not sure why.

This will take you to a download page with further instructions that, strangely, are only available on this download page. Scroll down a bit until you see something like this (for Windows):

Downloading Java JDK: Trying to Figure Out the Right Version

You may already have Java JDK installed. If not, this is where the official instructions are a bit off. The official instructions gave me a link to download either Open JDK or Oracle Java 8. I’m going open source, so I chose OpenJDK. This is the link they gave me:

https://openjdk.org/

You then have to – somewhat painfully – read through this link to find the right link. It is found here:

And here is the link they give you:

https://openjdk.org/projects/jdk/23/

From this link you have to painfully read to figure out which link you need.

Here is the one I found:

https://jdk.java.net/23/

Why didn’t they just send me directly there?

Because guess what? They sent you to JDK 23! Which is the wrong version for Neo4J! If you make the mistakes of following their instructions and that cascade of links and then try to run Neo4J it will work but you’ll get this warning:

Please use Java(TM) 17 or Java(TM) 21 to run Neo4j.

Yup! You need to downgrade to version 21 (at least) of the JDK to avoid this warning.

Also note that they sent us to a version of JDK without an installer. No biggie, I’ll give some instructions below on how to deal with that. But it would have been easier to have an installer.

Downloading the JDK: For Real This Time

Here are some better ways to find the right version of the JDK. Make sure you get a version between 17 and 21 and avoid all the pain above.

Links to download JDK Version 21 – preferably with an installer:

From Open Logic (includes installer!)
Oracle’s download and install instructions
Oracle’s download page

Setting Up Java_Home for Windows

If you didn’t use an installer (like me) then you will need to copy the downloaded JDK somewhere (I put it right off the root of the D drive) and then you’ll need to set the JAVA_HOME environment variable so that your Windows operating system knows where to find the JDK.

First run the “Edit the system environment variables” program:

Neo4J\_

You’ll see this screen below. Click “Environment Variables”:

You’ll then see this modal below. Click “New”:

Set the JAVA_HOME like this:

You may have to reboot in some cases.

Installing Neo4J

You previously (above) downloaded Neo4J. Now ‘install’ it by unzipping the neo4j-community-5.25.1-windows.zip file somewhere. I put it off the root D drive at this location:

D:\neo4j-community-5.25.1

Now (for Windows) run the command prompt (i.e. ‘cmd’) and navigate to the bin folder of the Neo4J install directory and run: (Using my directory as an example. Replace D:\neo4j-community-5.25.1 with where you installed yours.)

D:\neo4j-community-5.25.1\bin>neo4j console

You’ll see something like this:

Alternatively, you can run it as a service:

D:\neo4j-community-5.25.1\bin\neo4j install-service.

Running the Neo4J Console in the Browser

Now you are ready to run the Neo4J console in your browser by navigating in your browser to:

http://localhost:7474

You’ll see something like this:

The default login is username = neo4j and password = neo4j also. It will prompt you for a new password. I generated one at random. Be sure to save it or write it down.

Trying out Neo4J

There are two built-in tutorials that you can play with. I won’t go into all the detail but run this in the console at the top:

:play movie graph

The app will then walk you through the tutorial. The results are pretty amazing! For example, here is the graph playing “6 Degrees from Kevin Bacon” (well, actually 4 degrees but who is counting?)

And here is the shortest path from Kevin Bacon to Meg Ryan:

Go try to do that in your mundane relationship SQL database!

Conclusion

This post went over pretty much the same install instructions for Neo4J that you get from Neo4J themselves. I tried to add some value by correcting a few mistakes (i.e. sending you to the wrong version of the JDK) and clarifying a few rough spots in the instructions. I also went over a quick comparison of a relational SQL database vs a Graph Database and explained why you might want a graph database for an AI stack. I also did a quick comparison between Apache AGE vs Neo4J and why – for Windows at least – I’m currently going with Neo4J in our stack. (Though I hope to revisit Apache AGE in the future.)

Finally, here are a collection of helpful links in getting started with Neo4J.

Links

Neo4J Docs
General Install Instructions
Windows Specific Install Instructions
How to add Java-based User-defined procedures to Neo4J
Neo4J Developer Center
Cypher Graph Query Language Manual (i.e. graph database equivalent to SQL)
Import data
Profile queries, looking at the execution plan with EXPLAIN and PROFILE.
Get the official drivers for JavaScript, Java, .NET, and Python
Training:
- Take Online Training: Introduction to Graph Databases and Neo4j
- Watch Video Series: Intro to Graph Databases
- Graph Academy
- Other training
Neo4J Knowledge Builder: Extract Nodes and Relationships from Unstructured Data
- Llm-graph-builder.neo4jlabs.com
- Note: uses Neo4J Aura (paid solution in cloud, though there is also a free version for learning.)
- Local version on Github
Neo4J APOC Library: provides access to user-defined procedures and functions which extend the use of the Cypher query language into areas such as data integration, graph algorithms, and data conversion.
Learn more about the various types of data integrations possible with Neo4j.
Haystack Neo4J Integration (See also this link)
You can import your data from CSV files using the Cypher's LOAD CSV command.
Visualization Tool: Neo4j Bloom
Learn more about import in the Neo4j Developer Manual: Load CSV and Importing CSV Data into Neo4j.
Article on other open-source graph databases
Neo4J Desktop and here (See also here for download)
Neo4J Python drivers
Loading a graph database from a relational database
Cypher Cheat Sheet
Graph Databases Book (Free)
Other books

How to Remove a File From a GitHub Repo

Thu, 26 Dec 2024 12:00:00 -0700

Suppose you need to remove a file from a GitHub repo. How would you do that? If you just remove the file and check it in the older commits still contain that file. Which means it is still part of the overall repo. What to do?

According to GitHub itself you can use a tool like git-filter-repo to fix it.

I often find it helpful to use git as a way to compare text. If I’m working on an AI project, like the BookSearchArchive, I find I need to parse and reparse text until I’ve got it just right, or sometimes we’re dealing with copyrighted text. After I use git to compare versions of output, I can then use git-filter-repo to remove that part of the git history.

But git-filter-repo sure isn’t the easiest tool to figure out if you’re not deeply technical. Git-filter-repo is a tool that can do a lot more than just remove a single file, but that’s all I want to do with it and I don’t’ want to have to filter through a lot of documentation to figure that single use case out. So let me give you a simple set-by-step guide to using git-filter-repo to remove an unwanted file from your GitHub repo.

Main Use Case: Removing a Single File from History

First, don’t use your existing cloned repo. Instead, you need to download a mirror version of the repo that contains all branches and tags so that they can be rewritten. Something like this:

git clone --mirror https://github.com/pathofarchive/NameOfArchive.git

This will create a special version of your repo. It will be in a folder called “NameOfArchive.git” and will NOT contain any of your actual files. This is a bit unnerving I admit. But it is okay.

If you need to reassure yourself this is a real repo, do this:

git clone NameOfArchive.git test-clone

I’m assuming you run that command one level above the mirror repo, so you may have to “cd ..” first. If done correctly, this will create a new folder called ‘test-clone’ that will look like a correct version of your repo.

Okay, now you need to run git-filter-repo. To do this, you need to download this specific file into your mirror repo. The beauty of git-filter-repo is that you just drop this file into your repo and git-filter-repo is ‘installed’ and ready to go. Be sure there is no extension at the end of the file. (i.e. no ‘.txt’ etc.)

Now, if you are in Windows, you can probably run git-filter-repo like this using the python command:

python git-filter-repo --path file_to_remove.txt --invert-paths

Here ‘file_to_remove.txt’ is an example of a file that you are removing. --path specifies you’re going to give a file path. --invert-paths specifies that are wanting to keep everything else ‘as is’.

The process will run, and you’ll see something like this:

Parsed 239 commitsEnumerating objects: 858, done.
Counting objects: 100% (858/858), done.
Delta compression using up to 12 threads
Compressing objects: 100% (306/306), done.
Writing objects: 100% (858/858), done.
Selecting bitmap commits: 227, done.
Building bitmaps: 100% (107/107), done.
Total 858 (delta 547), reused 852 (delta 542), pack-reused 0

Next you need to push this all back up to your repo:

git push --mirror

And you’ll see something like this:

Enumerating objects: 858, done.
Writing objects: 100% (858/858), 7.03 MiB | 2.26 MiB/s, done.
Total 858 (delta 0), reused 0 (delta 0), pack-reused 858
remote: Resolving deltas: 100% (547/547), done.
To https://github.com/brucenielson/BookSearchArchive.git
 + 8d41638...1c4e354 main -> main (forced update)
 ! [remote rejected] refs/pull/1/head -> refs/pull/1/head (deny updating a hiden ref)
 ! [remote rejected] refs/pull/10/head -> refs/pull/10/head (deny updating a hdden ref)
 ! [remote rejected] refs/pull/11/head -> refs/pull/11/head (deny updating a hdden ref)

Etc.

The scary errors are to be expected.

You’ll likely find that you can’t easily update your local repo and that you need to get a fresh clone off of GitHub. So go do that and you should be all set.

AI Tutorial: Hybrid Search in Detail

Tue, 17 Dec 2024 12:00:00 -0700

Way back in this post, I conceptually explained how to do a “Hybrid” search where you combine the results of both a Lexical Search (i.e. word search) and a Semantic Search. It is time to make good on my promise to explain in detail how I did this.

The easiest way to do a Hybrid Search is to use the built-in Haystack DocumentJoiner component. Unfortunately, (and you’re probably sick of hearing me say this) there seems to be a bug in this component. I found that including it in my Haystack pipeline caused the pipeline to start streaming results from the LLM component before the documents were sent to the Large Language Model (LLM) component. It would first spit out a response ignoring all RAG docs sent and then later it will spit out the intended result. I’ll do a post on this problem later and see if it got fixed in later versions of Haystack.

So, we’re not going to do this the easy way for today. Instead, I’ll build a custom component that takes the results of a Lexical search and a Semantic search and combines them. Here is the custom component I wrote:

The Document / Query Collector

After receiving results from the Lexical and/or Semantic Search the results are all collected in another custom component that looks like this:

@component
class DocumentQueryCollector:
    def __init__(self, do_stream: bool = False, callback_func: Callable = None) -> None:
        self._do_stream: bool = do_stream
        self._callback_func: Callable = callback_func
    @component.output_types(documents=List[Document], query=str, llm_top_k=int)
    def run(self, query: str,
            llm_top_k: int = 5,
            semantic_documents: Optional[List[Document]] = None,
            lexical_documents: Optional[List[Document]] = None
            ) -> Dict[str, Any]:
        documents: List[Document] = []
        # Check for semantic documents vs lexical documents and, if both exist, merge them
        if semantic_documents is not None and lexical_documents is not None:
            # Combine semantic and lexical documents. But only include each document once and take highest scores first.
            output: List[Document] = []
            document_lists: List[list] = [semantic_documents, lexical_documents]
            docs_per_id: defaultdict = defaultdict(list)
            doc: Document
            for doc in itertools.chain.from_iterable(document_lists):
                docs_per_id[doc.id].append(doc)
            docs: list
            for docs in docs_per_id.values():
                # Take the document with the best score
                doc_with_best_score = max(docs, key=lambda a_doc: a_doc.score if a_doc.score else -inf)
                # Give a slight boost to the score for each duplicate - Add .1 to the score for each duplicate
                # but adjust the 0.1 boost by score of the duplicate
                if len(docs) > 1:
                    for doc in docs:
                        if doc != doc_with_best_score:
                            doc_with_best_score.score += min(max(doc.score, 0.0), 0.1)
                output.append(doc_with_best_score)
            output.sort(key=lambda a_doc: a_doc.score if a_doc.score else -inf, reverse=True)
            documents = output
        elif semantic_documents is not None:
            documents = semantic_documents
        elif lexical_documents is not None:
            documents = lexical_documents
        if self._do_stream:
            print()
            print("Retrieved Documents:")
            print_documents(documents)
        if self._callback_func is not None:
            self._callback_func()
        return {"documents": documents, "query": query, "llm_top_k": llm_top_k}

It is called and hooked into the pipeline like this:

doc_checker: DocumentQueryCollector = DocumentQueryCollector(do_stream=self._can_stream(),
                                                             callback_func=lambda: doc_collector_completed())
rag_pipeline.add_component("doc_query_collector", doc_checker)

Connect it to the semantic retriever:

rag_pipeline.connect("semantic_retriever.documents", "doc_query_collector.semantic_documents")

Connect it to the lexical retriever:

rag_pipeline.connect("lex_retriever.documents", "doc_query_collector.lexical_documents")

Connect it to the prompted builder:

rag_pipeline.connect("doc_query_collector.query", "prompt_builder.query")
rag_pipeline.connect("doc_query_collector.llm_top_k", "prompt_builder.llm_top_k")
rag_pipeline.connect("doc_query_collector.documents", "prompt_builder.documents")

We can invoke this as are starting node by calling it directly:

inputs: Dict[str, Any] = {
    "query_input": {"query": query, "llm_top_k": self._llm_top_k},
}
results: Dict[str, Any] = self._rag_pipeline.run(inputs, include_outputs_from=self._include_outputs_from)

The Power of Hybrid Search

Here is the key thing that really makes Hybrid search shine:

                # Give a slight boost to the score for each duplicate - Add .1 to the score for each duplicate
                # but adjust the 0.1 boost by score of the duplicate
                if len(docs) > 1:
                    for doc in docs:
                        if doc != doc_with_best_score:
                            doc_with_best_score.score += min(max(doc.score, 0.0), 0.1)

What I’m doing here is I’m checking if there are any documents that were returned by both the lexical and semantic search. If so, I allow for a small boost to the score of that document.

Now typically (as discussed in this post) a search will either score well Lexically OR it will score well Semantically. What I mean is that if I search on ‘induction’ (a single word) the Lexical search will probably score well. But if I search on ‘What is induction and how does it relate to testability?’ the Lexical search will do poorly but the semantic search will do well.

But if we notice that a document shows up for both, that is still a good sign. So, I allow a boost of up to 0.1 for showing up on both searches. Imagine I get a semantic score of say 0.89 and a Lexical score of 0.01. I’d then add those scores together for a 0.90. (I max out the boost at 0.1 to avoid strange results far over a score of 1.0. But it would be rare that would happen.)

Conclusion

And that is how I built a custom hybrid search for the Book Search Archive. There is not much to it. You just do both searches and concatenate them together with a slight boost if there are duplicates.

In a future post I’ll cover how to use a Re-Ranker to then sort these results for the Large Language Model.

A Local Text-to-Speech Model Using Suno Bark

Wed, 11 Dec 2024 08:55:00 -0700

In our last post, we looked at how to stream a text-to-speech model using the Hugging Face API. We had mixed results at best. So, in this post, I’ll cover how to do text-to-speech (TTS) using a local TTS model.

Introducing Suno Bark… Again

In our last post I tried out several different models via the Hugging Face API including suno/bark and suno/bark-small. Both of these are excellent, but relatively small and manageable, open-sourced text-to-speech models available on Hugging Face. I’ll be working with Suno Bark Small, but it is easy enough to pass in Suno Bark or some other text-to-speech model, if you desire.

Suno Bark Small has the main features I’m looking for. It is simple, it doesn’t require additional installs (unlike parler-tts) and most importantly it has voices you can use that stay consistent.

Many text-to-speech models sound different every time you use them. You just get a random voice. It might be a man one time and a woman another. Suno Bark Small has voices built in that you can specify that sound the same every time.

As it turns out, you can even specify a voice with an accent. For the BookSearchArchive I decided to give the voice a German accent by utilizing a German voice but giving it English text.

The TextToSpeechLocal Class

I built a TTS custom Haystack component for you to use called TextToSpeechLocal that can be found in the customhaystackcomponents.py file. (Here is the code base at the time of this Blog post. Most recent code found here.)

Let’s start with declaring the class and creating an initialization method:

@component
class TextToSpeechLocal:
    def __init__(self, model_name_or_path: str = "suno/bark-small"):
        # Initialize the processor
        self.device = "cuda" if torch.cuda.is_available() else "cpu"
        self.processor = AutoProcessor.from_pretrained(model_name_or_path, torch_dtype=torch.float16)
        self.model = BarkModel.from_pretrained(model_name_or_path, torch_dtype=torch.float16).to(self.device)

You can see that I allow you to pass a model name and default to suno/bark-small. However, note that probably no other model will work as of today other than suno/bark itself. You’ll see why this is true in a moment. Note that we are using the Hugging Face AutoProcessor and ‘from_pretrained’ methods to create the model.

Next let’s create a method for playing audio:

@staticmethod
def _play_audio(audio_data: np.ndarray, sample_rate: int = 24000) -> None:
    audio_data = audio_data.astype("float32")
    sd.play(audio_data, samplerate=sample_rate)
    sd.wait()  # Wait until the audio finishes playing

This is simple enough. You pass in a numpy array with audio_data and give it a sampling rate. I then make sure it’s in a float32 format (I’ll explain why this is important later) and then uses the SoundDevice library to play it. Then it ‘waits’ for the audio to finish.

Of course, the ‘run’ method is where all the real work happens. We will play in a string that (presumably) is created by the Large Language Model (LLM) and it will chop the string up into sentences and then play each sentence:

@component.output_types(text=str)
def run(self, reply: str) -> Dict[str, Any]:
    # Split the input text into sentences using regular expression
    sentences: List[str] = re.split(r'(?<=[.!?])\s+', reply.strip())

    # Process each sentence
    sentence: str
    for sentence in sentences:
        # Use the v2/de_speaker_0 voice preset
        voice_preset: str = "v2/de_speaker_0"

        # Prepare the inputs for the model
        inputs: dict = self.processor(sentence,
                                      voice_preset=voice_preset,
                                      return_tensors="pt",
                                      return_attention_mask=True)

        # Ensure inputs are moved to the correct device
        inputs = {key: value.to(self.device) for key, value in inputs.items()}

        audio_array = self.model.generate(**inputs).to(self.device)
        audio_array = audio_array.cpu().numpy().squeeze()

        # Play the generated audio immediately
        self._play_audio(audio_array)

    # After all sentences are processed, return the last audio chunk and the full text
    return {"text": reply}

The input is a string called ‘reply’ (from the LLM) and the output of this component is a string called ‘text’. I use a regular expression to split up the text into sentences by looking for punctuation like ‘.’, ‘!’, or ‘?’. Then I feed each sentence to the ‘processor’ created in the init method:

        inputs: dict = self.processor(sentence,
                                      voice_preset=voice_preset,
                                      return_tensors="pt",
                                      return_attention_mask=True)

This takes each sentence and turns it into an appropriate list of inputs to be fed into the actual model. The inputs will include an attention mask, encodings, etc.

Finally, we generate the actual audio using the model:

audio_array = self.model.generate(**inputs).to(self.device)

And then we squeeze it down to a vector (single dimensional array) to be sure it is in the right format for SoundDevice.

audio_array = audio_array.cpu().numpy().squeeze()

Finally, we call our function to play the audio. On my laptop, the result is pretty slow. If you have a GPU it will probably be able to keep up. But in a future post I’ll explore ways to make this run better.

Voice Presets

You may have noticed I set the voice to "v2/de_speaker_0". This is obviously meant for a German speaker, not an English speaker. But I thought it would be cool to give the BookSearchArchive a German accent. I wasn’t sure if a German model could speak English nor was I sure that if it could it would speak with a German Accent. But it could read English fine, and I think it has a nice German accent.

Specifying a specific voice is how you get Suno Bark to use a consistent voice. Also note that I currently hard coded a Suno Bark Small voice and if you tried to use a different model it would likely not know what to do with this voice preset. I need to fix that in a future version to allow any TTS model.

There is one problem with using a preset voice like this: this ‘speaker’ is tied specifically to suno/bark (though it also works on suno/bark-small). So, this custom component will fail on any model you pass in that doesn’t specifically have this voice. Which is, presumably, almost all of them. So, this is really a suno bark specific component. I will come up with a way to fix that in the future and make it more generic.

You can find a list of voice presets here.

Using TTS Component in Your Pipeline

Here is my revised pipeline using the TTS component:

That is looking pretty crazy, isn’t it? I wrote a new “MergeResults” component that takes the documents list from the doc_query_ collector as well as the results from the LLM and collates them all together to make it easy to find the final results. I then connect the ‘reply’ from that component to the Text-to-Speech (TTS) component.

This is all probably more complicated it then it needed to be, but it works. I’ll clean it up later. But this allows me to only connect in the TTS component if the user asks for it via the new ‘use_voice’ parameter:

if self._use_voice and not self._can_stream():
    # Add the text to speech component
    tts_node = TextToSpeechLocal()
    rag_pipeline.add_component("tts", tts_node)
    rag_pipeline.connect("merger.reply", "tts.reply")

I know that no matter what other parameters are set, the ‘merger’ will always have what I need.

Conclusions

This code will add a voice to your Haystack pipeline using a local model and allow your LLM to speak to you. We covered how to give it a preset voice so that the model speaks with a consistent voice and also how to give it an accent.

Links:

Testing Hugging Face Serverless Text-To-Speech Models

Wed, 04 Dec 2024 08:55:00 -0700

I’d like to add a voice to BookSearchArchive, my test project for the Mindfire open-source AI stack, which I first discussed in this post. I thought it would be fun to let the Large Language Model (LLM) you converse with be able to actually chat with you via audio.

There are two possible ways to do this. One is we could run a local text-to-speech model. The other is we could use the Hugging Face API. (*)

Here is a link to the code at the time of this blog post. The most up-to-date code is found here.

Local vs Serverless API Text-To-Speech

Since my laptop has a sucky GPU, my preference would be to stream a Text-To-Speech model on the Hugging Face rate-limited Serverless API – similar to how we did for the BookSearchArchive’s LLM. (*)

Unfortunately, things didn’t go as planned.

The Hugging Face rate-limited server does have some Text-To-Speech models available on it. My first attempt was to just try the default which is suno/bark. It turns out this model is available on the Hugging Face serverless API, but according to the error message I received back, it is reserved for paying Pro customers.

Bad request:

Model requires a Pro subscription; check out hf.co/pricing to learn more. Make sure to include your HF token in your query.

I also tried out suno/bark-small but it always seems to return an internal server error:

Error generating audio for suno/bark-small: 500 Server Error: Internal Server Error for url: https://api-inference.huggingface.co/models/suno/bark-small (Request ID: OaHuye9GkoJeQqir0WW4d)

You can tell that these models exist on the Hugging Face server because you can navigate to them through your browser and see them for yourself. (Try: suno/bark, suno/bark-small, microsoft/speecht5_tts and you can see the server returns a proper result). I can understand why suno/bark doesn’t work since it requires a paid subscription, but why do you get an error on suno/bark-small?

After doing some research, it turns out that the Hugging Face server has setup suno/bark-small incorrectly: (Found on the Hugging Face discussion board)

“The Serverless Inference API is currently turned off for most models. This is probably because the shared resources on the server have been exhausted. It is turned on for microsoft/speecht5_tts, but it does not actually work because the configuration settings are not properly set in README.md or json.”

Well, that’s disappointing.

Finding a Working Serverless Model

I wrote some code to try out several different models on the Hugging Face server and attempt to generate text-to-speech. By trying out several models at once, I can see if any of them are setup correctly to work.

Here is my test code:

from huggingface_hub import InferenceClient
import requests
from pathlib import Path

# Load Hugging Face API secret - Put the secret in a text file and read it
hf_secret = open(r'D:\Documents\Secrets\huggingface_secret.txt', 'r').read().strip()


def get_model_details(model_id: str, token: str):
    """Fetch model details, including sample rate, using the Hugging Face API."""
    url = f"https://huggingface.co/api/models/{model_id}"
    headers = {"Authorization": f"Bearer {token}"}
    try:
        response = requests.get(url, headers=headers)
        print(f"\n>>> Retrieving details for model: {model_id}")
        print(f"Status Code: {response.status_code}")
        if response.status_code == 200:
            model_info = response.json()
            # Try to extract the sample rate from the metadata
            # sample_rate = model_info.get("config", {}).get("sample_rate", None)
            # print(f"Sample Rate: {sample_rate}")
            return {
                "id": model_info.get("id"),
                "modelType": model_info.get("modelType"),
                "pipeline_tag": model_info.get("pipeline_tag"),
                "library_name": model_info.get("library_name"),
                # "sample_rate": model_info.get("config", {}).get("sampling_rate", 16000),
            }
        else:
            print(f"Failed to retrieve details for {model_id}.")
            return None
    except Exception as e:
        print(f"Error fetching model details for {model_id}: {e}")
        return None


def generate_audio(model_id: str, token: str, text: str):
    """Generate audio using InferenceClient."""
    client = InferenceClient(api_key=token)
    try:
        print(f"\n>>> Generating audio with model: {model_id}")
        audio_data = client.text_to_speech(text, model=model_id)
        # model_details = get_model_details(model_id, token)
        # config = AutoConfig.from_pretrained(model_id)
        if isinstance(audio_data, bytes):
            file_name = model_id.replace("/", "_")
            audio_file = Path(f"{file_name}_test_sentence.flac")
            audio_file.write_bytes(audio_data)
            print(f"Audio saved to {audio_file}")
            return True
        else:
            print(f"Unexpected response type from {model_id}.")
            return False
    except Exception as e:
        print(f"Error generating audio for {model_id}: {e}")
        return False


def try_models(models, text, token):
    """Test models and generate a summary report."""
    results = {}
    for model in models:
        print("\n" + "=" * 80)
        print(f"Processing model: {model}")
        print("=" * 80)

        model_details = get_model_details(model, token)
        if model_details:
            print("Model Details:")
            print(model_details)
        else:
            print(f"Skipping {model} due to missing details.")

        success = generate_audio(model, token, text)
        results[model] = "Success" if success else "Failed"

    print("\n" + "=" * 80)
    print("SUMMARY REPORT")
    print("=" * 80)
    for model, result in results.items():
        print(f"Model: {model}\n  Result: {result}")
    print("=" * 80)


if __name__ == "__main__":
    models = [
        "suno/bark",
        "suno/bark-small",
        "facebook/mms-tts-eng",
        "microsoft/speecht5_tts",
    ]
    text = "Hello, welcome to the world of text to speech!"
    try_models(models, text, hf_secret)

This code runs through several models and tries them all out:

In truth, I tried a lot more models. Most weren’t on the server at all. The 4 above all exist on the server but most of them throw an internal server error. My code tests if they are there, tries to get info on them, and then tries to create a file using Hugging Face’s InferenceClient method.

“get_model_details” is a function that gets details on the model – if it exists – and prints out information about the model.

“try_models” is a function that actually tries to call the models using Hugging Face’s InferenceClient method.

My code shows that all four of these exist on the Hugging Face server, but bark-small and speecht5-tts both get internal server errors. Only facebook/mms-tts-eng actually produces an audio file. (Called facebook_mms-tts-eng_test_sentence.flac).

Model: suno/bark
  Result: Failed
Model: suno/bark-small
  Result: Failed
Model: facebook/mms-tts-eng
  Result: Success
Model: microsoft/speecht5_tts
  Result: Failed

Environment Setup

To make this code work I upgraded tokenizer and transformer and installed SoundDevice. (PyPI SoundDevice page.) You can see the updated environment in the requirements.txt file. Try running:

pip install tokenizers==0.19.1

pip install transformers==4.43.2

pip install sounddevice

So, if you want to use the Hugging Face serverless Api it looks like facebook/mms-tts-eng is the only option I was able to find for you.

Playing Audio Immediately

Of course, my code saves the audio out to a file rather than actually playing the audio directly. You could then play the audio from the file, of course. But that’s not the most useful. Unfortunately, the facebook/mms-tts-eng model seems to (at least by default) create flac files which are compressed. So you can’t play them via the SoundDevice library directly. I will work out how to deal with that problem and publish a solution in a future post.

Using a Local Model

Alternatively, you could just use a local model. If you check my latest code, I do have a working version of a local text-to-speech model. You can activate it via the RagPipeline method’s new parameter:

use_voice=True

Checkout the TextToSpeechLocal class in customhaystackcomponents.py. I’ll cover this code in detail in a future post.

Update: The problem with the facebook/mms-tts-eng turned out to be that the Hugging Face server set it up to return flac files and currently soundfile (which underlies most libraries in python that play audio) has a bug that it won't play flac files correctly. See this bug I reported complete with how to simply replicate the problem.

Notes

(*) Recall (back in this post) we used Haystack’s HuggingFaceAPIGenerator component and used the HuggingFaceH4/zephyr-7b-alpha model, which has been optimized for Hugging Face’s serverless API.

Links:

Suno Bark on Github
Suno Bark on Hugging Face
Suno Bark Small on Hugging Face
Test out Suno Bark in Hugging Face Spaces
Hugging Face Inference Client Documentation:
- Inference Client Documentation
- InferenceClient method
Hugging Face Serverless API
Hugging Face Discuss Board
- Discussion about Suno/Bark-Small model not working: https://discuss.huggingface.co/t/undefined-error-on-inference-api-serverless-for-several-hf-text-to-speech-tasks/79230/3

PDFs vs HTML: The Importance of Metadata for Retrieval Augmented Generation

Tue, 19 Nov 2024 08:55:00 -0700

Up to this point, we’ve been loading EPUB documents instead of PDFs, and I had a good personal reason for that. EPUBs are essentially HTML files, so (as discussed in this post) the key to loading EPUBs—since they aren’t built into Haystack—is treating them as HTML. However, in the latest release of my code I’ve also added support for loading PDFs from the document store database.

The Advantages of HTML

The real advantage of coding for HTML documents—whether EPUBs or actual web pages—is that I can later reuse my code for web pages. Additionally, since HTML header tags provide valuable metadata, I use them to generate richer metadata.

Take, for example, the document I retrieved from my Karl Popper document store (as discussed in the last post):

Document 7: 
Score: 0.05 
Item #: 17 
Page #: 205 
Item Id: Ch05 
Book Title: Conjectures and Refutations 
Paragraph #: 69 
Section Name: XII 
Chapter Title: 5 Back to the Presocratics 
Content: This, I believe, is the true theory of knowledge (which I wish to 
submit for your criticism): the true description of a practice which arose in 
Ionia and which is incorporated in modern science (though there are many 
scientists who still believe in the Baconian myth of induction): the theory that 
knowledge proceeds by way of conjectures and refutations.

Notice how this document fragment includes a page number, book title, chapter title, and even section title. When I’m using my “Karl Popper Archive” (as discussed in the previous post) for research, having metadata like this is incredibly helpful. It also allows me to have answers to questions from a Large Language Model (LLM) cite their sources. Rich metadata like this is typically not available in most PDFs—or at least not without complex, custom data processing. But by reading header tags in HTML, I was able to extract this metadata directly from the document as I parsed it.

This is why I prefer EPUBs over PDFs: they provide an easy, rich source of metadata.

The Advantages of PDFs

However, EPUBs aren’t exactly a common document format. For instance, not every book by Karl Popper is available as an EPUB. As a result, my archive also needs to be able to load PDF files. PDFs are such a widely used format for documents, manuals, and tables of useful data, so we need a way to convert PDF files into document fragments that we can store in the archive as well.

How to Load PDFs: The Simple Way

Loading PDFs is actually built into Haystack! Initially, I used the built-in Haystack component to load PDFs into the Book Search Archive, and you can see an example of that in this particular commit of my code. However, I later decided I wasn't satisfied with the results and believed I could improve on it, so I replaced the built-in component with a custom Haystack solution that loads PDFs using (currently) PyPDF. This gave me more control over how the document fragments and related metadata were extracted from the PDFs. I’ll cover my custom approach in a future post. Even with this improved version, it’s still not exactly what I want, so I’ll likely move away from PyPDF at some point and explore a more fine-grained solution.

But for now, let’s go over how to use the built-in Haystack component. Below is a revised version of my document conversion pipeline that uses the PyPDFToDocument component built into Haystack:

As you can see, I’ve set up two parallel pipes in my pipeline. The “epub_vs_pdf_splitter” takes a list of file paths and routes the EPUB books to a pipeline for EPUBs (using the “html_converter” we discussed in previous posts) and the PDFs to a pipeline with the PyPDFToDocument converter built into Haystack. Then, the “epub_pdf_merger” combines everything back into a final list of documents, which are then passed through the remaining part of the pipeline for cleaning, splitting into document fragments, etc.

Pretty simple, right?

But how do you actually create the custom components needed to perform this kind of split?

Splitting and Merging

Here is the code that creates this pipeline:

@component 
class EpubVsPdfSplitter: 
    @component.output_types(epub_paths=List[str], pdf_paths=List[str]) 
    def run(self, file_paths: List[str]) -> Dict[str, List[str]]: 
        epub_paths: List[str] = [] 
        pdf_paths: List[str] = [] 
        for file_path in file_paths: 
            if file_path.lower().endswith('.epub'): 
                epub_paths.append(file_path) 
            elif file_path.lower().endswith('.pdf'): 
                pdf_paths.append(file_path) 
            else: 
                raise ValueError(f"File type not supported: {file_path}") 
        return {"epub_paths": epub_paths, "pdf_paths": pdf_paths}

Not much here. It takes in a list of strings as file paths and outputs that list separated into a list of pdf paths vs a list of epub paths.

@component 
class EPubPdfMerger: 
    @component.output_types(documents=List[Document]) 
    def run(self, epub_docs: List[Document], pdf_docs: List[Document]) -> Dict[str, List[Document]]: 
        documents: List[Document] = [] 
        for doc in epub_docs: 
            documents.append(doc) 
        for doc in pdf_docs: 
            documents.append(doc) 
        return {"documents": documents}

And the above component then takes a list of epub Haystack Documents (i.e. the Document class) and a list of pdf Haystack Documents and merges them into a final combined list of Haystack Documents. Again, very simple.

Finally, we just need to connect it all together like this:

        doc_convert_pipe: Pipeline = Pipeline()     
        doc_convert_pipe.add_component("epub_vs_pdf_splitter", EpubVsPdfSplitter()) 
        doc_convert_pipe.add_component("pdf_loader", PyPDFToDocument()) 
        doc_convert_pipe.add_component("epub_loader", EPubLoader(verbose=self._verbose)) 
        doc_convert_pipe.add_component("html_parser", 
                                       HTMLParserComponent(min_paragraph_size=self._min_paragraph_size, 
                                                           min_section_size=self._min_section_size, 
                                                           verbose=self._verbose)) 
        doc_convert_pipe.add_component("html_converter", HTMLToDocument()) 
        doc_convert_pipe.add_component("epub_pdf_merger", EPubPdfMerger()) … 
        doc_convert_pipe.connect("epub_vs_pdf_splitter.epub_paths", "epub_loader.file_paths") 
        doc_convert_pipe.connect("epub_vs_pdf_splitter.pdf_paths", "pdf_loader.sources") 
        doc_convert_pipe.connect("epub_loader.html_pages", "html_parser.html_pages") 
        doc_convert_pipe.connect("epub_loader.meta", "html_parser.meta") 
        doc_convert_pipe.connect("html_parser.sources", "html_converter.sources") 
        doc_convert_pipe.connect("html_parser.meta", "html_converter.meta") 
        doc_convert_pipe.connect("pdf_loader.documents", "epub_pdf_merger.epub_docs") 
        doc_convert_pipe.connect("html_converter.documents", "epub_pdf_merger.pdf_docs") 
        doc_convert_pipe.connect("epub_pdf_merger.documents", "remove_illegal_docs") … 
        self._doc_convert_pipeline = doc_convert_pipe

Don't forget to run:

pip install -r requirements.txt

If you need help with environment setup, this blog post will help.

To get Haystack’s PyPDFToDocument component to work you’ll probably need to do:

pip install pypdf

Conclusions

And that’s the simplest approach to adding PDF documents into our Book Search Archive. I’ll cover my improved more custom approach in a future post as well as explore other alternatives, such as the built-in PDFMinerToDocument component.

Implementing a Lexical Search

Fri, 15 Nov 2024 08:55:00 -0700

In my last post, I went over the idea of a hybrid search where we merge the results of both a lexical search (i.e. a traditional keyword search) and a semantic search together into a single result. The end result was better than either individually because we could give a bonus for a passage being a good lexical and semantic search.

Let’s now talk about how to actually implement a Lexical Search. You can find the version of the code at the time of this blog post here. If you want the most up-to-date version of the code that is found here.

Lexical, Semantic, and Hybrid Search

I’ve rewritten my “Book Search Archive” to have three options for searching. Lexical, Semantic, or Hybrid:

class SearchMode(Enum):
    LEXICAL = 1
    SEMANTIC = 2
    HYBRID = 3

You can now specify which search to you be simply changing a parameter in the RagPipeline instance upon initializing it, like this:

    rag_processor: RagPipeline = RagPipeline(table_name="book_archive",
                                             generator_model=model,
                                             postgres_user_name='postgres',
                                             postgres_password=postgres_password,
                                             postgres_host='localhost',
                                             postgres_port=5432,
                                             postgres_db_name='postgres',
                                             use_streaming=True,
                                             verbose=False,
                                             llm_top_k=5,
                                             retriever_top_k_docs=None,
                                             include_outputs_from=include_outputs_from,
                                             search_mode=SearchMode.HYBRID,
                                             embedder_model_name="BAAI/llm-embedder")

Please that with SearchMode.LEXICAL or SearchMode.SEMANTIC as you see fit.

The Lexical Search In Detail

My Retrieval Augmented Generation (RAG) pipeline now looks like this:

You can see that we take a query string and an integer telling how many ‘top’ k results to take and it is sent off to both the semantic search part of the pipeline as well as the lexical search part of the pipeline.

The ‘lex_retriever’ node looks like this:

        # Add the retriever component(s) depending on search mode
        if self._search_mode == SearchMode.LEXICAL or self._search_mode == SearchMode.HYBRID:
            lex_retriever: RetrieverWrapper = RetrieverWrapper(
                PgvectorKeywordRetriever(document_store=self._document_store, top_k=self._retriever_top_k))
            rag_pipeline.add_component("lex_retriever", lex_retriever)
            rag_pipeline.connect("query_input.query", "lex_retriever.query")
            rag_pipeline.connect("lex_retriever.documents", "doc_query_collector.lexical_documents")

Notice that the underlying component is a custom component called “RetrieverWrapper” into which I pass a built-in Haystack component called “PgvectorKeywordRetriever”. PgvectorKeywordRetriever is doing all the real work. It simply uses the built-in ability in PostgreSQL to do keyword searches within a string contained in the database.

Under the hood it uses the tsquery command in PostgreSQL to do the search. We can emulate this directly in PgAdmin (see this post for details) by using a query like this:

SELECT ts_rank_cd(to_tsvector('english', content), to_tsquery('english', 'induction')) AS rank
    , content
        , meta->>'book_title' AS book_title
        , meta->>'section_title' AS section_title
    , *  
FROM popper_archive
WHERE to_tsvector('english', content) @@ to_tsquery('english', 'induction')
ORDER BY rank DESC;

The above query will query data out of the meta data in my document store. Notice how I’m looking for all cases of the word ‘induction’ (keeping with my Karl Popper theme) both in the content field as well as the ‘book_title’ and ‘section_title’ meta data fields contained as a JSON in the meta field. I would note that if you need to do fancy queries this is a good example of how to write your own customized queries and then use Psycopg to run the custom queries. The PgvectorKeywordRetriever simply uses the tsquery functionality to do a keyword search.

RetrieverWrapper for Streaming Retrieved Results

So why do I wrap the PgvectorKeywordRetriever component in a custom component called RetrieverWrapper? Well, let’s take a look at the code for this custom component:

@component
class RetrieverWrapper:
    def __init__(self, retriever: Union[PgvectorEmbeddingRetriever, PgvectorKeywordRetriever],
                 do_stream: bool = False) -> None:
        self._retriever: Union[PgvectorEmbeddingRetriever, PgvectorKeywordRetriever] = retriever
        self._do_stream: bool = do_stream
        # Alternatively, you can set the input types:
        # component.set_input_types(self, query_embedding=List[float], query=Optional[str])

    @component.output_types(documents=List[Document])
    def run(self, query: Union[List[float], str]) -> Dict[str, Any]:
        documents: List[Document] = []
        if isinstance(query, list):
            documents = self._retriever.run(query_embedding=query)['documents']
        elif isinstance(query, str):
            documents = self._retriever.run(query=query)['documents']
        if self._do_stream:
            print()
            if isinstance(self._retriever, PgvectorEmbeddingRetriever):
                print("Semantic Retriever Results:")
            elif isinstance(self._retriever, PgvectorKeywordRetriever):
                print("Lexical Retriever Results:")
            print_documents(documents)
        # Return a dictionary with documents
        return {"documents": documents}

When you initialize the component it takes another component – either PgvectorEmbeddingRetriever or PgvectorKeywordRetriever and a Boolean value to stream the results or not stream the results. Then, when you run the component, it calls the component you passed to do the real work. But it then prints those results out to the console before it passes the results on to the next node in the RAG pipeline. The idea is that we get to view the retrieved documents / paragraphs before the Large Language Model (LLM) receives them so that we don’t have to wait for the LLM to generate a response. In other words, RetrieverWrapper is a streaming version of whichever retriever we decide to wrap.

Note also that I also set a streaming option for the LLM. But I’ll cover this in a future post.

The Query Input Component

You might notice that everything starts with the ‘query_input’ component. To be honest I just got tired of making changes to my pipelines and changing which node was the start of the pipeline. On top of that, as I drew parallel pipelines it was inconvenient to try to have two starts that I had to invoke at once. So, I made a single custom component like this:

@component
class QueryComponent:
    @component.output_types(query=str, llm_top_k=int)
    def run(self, query: str, llm_top_k: int) -> Dict[str, Any]:
        return {"query": query, "llm_top_k": llm_top_k}

This takes a query string and a number of ‘top’ k documents to send to the LLLM. And then it passes it along to the appropriate components. You set it up like this:

rag_pipeline.add_component("query_input", QueryComponent())

Then connect it to all the appropriate places in the pipeline (depending on what kind of search you are doing) like this…

Connection to Semantic Search:

rag_pipeline.connect("query_input.query", "query_embedder.text")

Sent to the Lexical Retriever:

rag_pipeline.connect("query_input.query", "lex_retriever.query")

To the Document Query Collector (so that it receives the query and top k input parameter):

rag_pipeline.connect("query_input.query", "doc_query_collector.query")
rag_pipeline.connect("query_input.llm_top_k", "doc_query_collector.llm_top_k")

I’ll cover the other custom component, the “document / query collector”, in a future post as it is more related to the details of the Hybrid search. But what it does is takes the parallel searches (lexical and semantic) as well as the query input parameters and joins the results all together to be passed to the Large Language Model.

Conclusions

Lexical Search is built-into both PostgreSQL and Haystack. So it is easy to implement Lexical searching for our Book Search Archive. Semantic Search is great, but sometimes you want to search for a specific word or two rather than the semantic meaning of a sentence or question. So having Lexical search as part of our Book Search Archive was an important feature to add. We also covered how to use the power of lexical search using SQL queries in PostgreSQL for building custom queries. We also showed how to stream the results of the query if you want to view it before it goes to the LLM for processing.

Hybrid Search for Retrieval Augmented Generation

Fri, 08 Nov 2024 12:00:00 -0700

In my last post I introduced the “Book Search Archive.” You can find the repository for this code here. And the codebase as it was at the time of this blog post here. (i.e., if you want to recreate what is in this blog post, use the link tied to this specific post. If you want to see the latest version of my code, use the first link).

Now, let’s talk about some of the improvements I built into my new code release for the Book Search Archive. One of these is a custom-built ‘hybrid search.’ What’s a hybrid search? Glad you asked!

Semantic Search vs Lexical Search

I've extensively discussed semantic search in previous posts. If you need a primer, see the links section below.

Simply put, semantic search retrieves results based on meaning rather than exact keywords. For example, using the Book Search Archive to search across several works by philosopher Karl Popper, I might ask, "Is induction a myth?" Now, Karl Popper—widely regarded as one of the greatest philosophers to ever live—asserted that induction is indeed a myth. Let's see how well our semantic search archive surfaces relevant results.

To set this up, I specify semantic search in the RAG pipeline configuration, like this:

    rag_processor: RagPipeline = RagPipeline(table_name="popper_archive",
                                             generator_model=model,
                                             postgres_user_name='postgres',
                                             postgres_password=postgres_password,
                                             postgres_host='localhost',
                                             postgres_port=5432,
                                             postgres_db_name='postgres',
                                             use_streaming=True,
                                             verbose=False,
                                             llm_top_k=5,
                                             retriever_top_k_docs=None,
                                             include_outputs_from=include_outputs_from,
                                             search_mode=SearchMode.SEMANTIC,
                                             embedder_model_name="BAAI/llm-embedder")

Then set the query and run it:

query: str = "Is induction a myth?"
rag_processor.generate_response(query)

Here is the top result returned from a broad search across Karl Popper's works:

Document 1:
Score: 0.8961648406884185
Item Id: Ch01
Item Num: 2
Book Title: Conjectures and Refutations
Page Number: 71
Section Name: VIII
Chapter Title: 1 Science: Conjectures and Refutations
Paragraph Num: 78
Content: I may summarize some of my conclusions as follows: (1) Induction, i.e.
inference based on many observations, is a myth. It is neither a psychological
fact, nor a fact of ordinary life, nor one of scientific procedure. (2) The
actual procedure of science is to operate with conjectures: to jump to
conclusions—often after one single observation (as noticed for example by Hume
and Born).

Not bad, right? This result directly addresses our question, doesn’t it?

Of course, we might have found that result without semantic search by simply searching for specific keywords. This approach is known as a "Lexical Search." Let’s try it by adjusting our pipeline as follows:

    rag_processor: RagPipeline = RagPipeline(table_name="popper_archive",
                                             generator_model=model,
                                             postgres_user_name='postgres',
                                             postgres_password=postgres_password,
                                             postgres_host='localhost',
                                             postgres_port=5432,
                                             postgres_db_name='postgres',
                                             use_streaming=True,
                                             verbose=False,
                                             llm_top_k=5,
                                             retriever_top_k_docs=None,
                                             include_outputs_from=include_outputs_from,
                                             search_mode=SearchMode.LEXICAL,
                                             embedder_model_name="BAAI/llm-embedder")

Now, simplify the query to just the two keywords:

query: str = "induction myth"

This time, the well-matched paragraph that directly answers our question doesn’t even appear in the top 5 results! Here’s my new top result:

Document 1:
Score: 0.10909091
Item Id: ch32
Item Num: 32
Book Title: Unended Quest
Page Number: 163
Chapter Title: 32. Induction; Deduction; Objective Truth
Paragraph Num: 1
Content: There is perhaps a need here for a few words about the myth of
induction, and about some of my arguments against induction. And since at
present the most fashionable forms of the myth connect induction with an
untenable subjectivist philosophy of deduction, I must first say a little more
about the objective theory of deductive inference, and about the objective
theory of truth.

From this result, you can still infer that Karl Popper considered induction a myth, but the answer is less direct. This makes sense, as we were searching for "induction" and "myth," so it returned the paragraph with the highest occurrence of those words rather than truly answering the question.

Sometimes, though, it’s helpful to search by specific words, while other times a semantic search is more effective. The Book Search Archive repository allows you to easily switch between a traditional word-based search (i.e., a lexical search) and an AI-driven semantic search using text embeddings and language models.

But what if you’re unsure which is best? Why not use both at once?

Hybrid Search

Ideally, we want to perform both a lexical and semantic search, then take the best results from each.

I've modified our Haystack pipeline to enable this. The top portion of the Retrieval-Augmented Generation (RAG) pipeline now looks like this:

Notice that we now have both a ‘semantic_retriever’ and a ‘lex_retriever’ component in the pipeline. The results from both are sent to the ‘doc_query_collector’ component, which merges them.

In a future post, I’ll cover the best practices for merging results and explain how I coded the ‘doc_query_collector’ component. For now, assume I rank the results from both searches and sort them together.

I've found hybrid search to be a powerful approach. When searching by a keyword or two, the ‘lex_retriever’ generally outperforms the ‘semantic_retriever,’ so its results rise to the top. However, when I pose a longer question, the ‘semantic_retriever’ usually outshines the ‘lex_retriever,’ bringing its results to the forefront. This hybrid approach really does deliver the best of both worlds.

To demonstrate this, set up the RAG pipeline like this:

rag_processor: RagPipeline = RagPipeline(table_name="popper_archive",
                                         generator_model=model,
                                         postgres_user_name='postgres',
                                         postgres_password=postgres_password,
                                         postgres_host='localhost',
                                         postgres_port=5432,
                                         postgres_db_name='postgres',
                                         use_streaming=True,
                                         verbose=False,
                                         llm_top_k=5,
                                         retriever_top_k_docs=None,
                                         include_outputs_from=include_outputs_from,
                                         search_mode=SearchMode.HYBRID,
                                         embedder_model_name="BAAI/llm-embedder")

First, try the query "Is induction a myth?" Interestingly, this is now my top result:

Document 1:
Score: 0.9725419972234968
Item Id: ch32
Item Num: 32
Book Title: Unended Quest
Page Number: 171
Chapter Title: 32. Induction; Deduction; Objective Truth
Paragraph Num: 23
Content: But this was to be expected. Since there can be no theory-free
observation, and no theory-free language, there can of course be no theory-free
rule or principle of induction; no rule or principle on which all theories
should be based. Thus induction is a myth. No “inductive logic” exists. And
although there exists a “logical” interpretation of the probability calculus,
there is no good reason to assume that this “generalized logic” (as it may be
called) is a system of “inductive logic”.

Notice that the hybrid search provided an even more straightforward answer to our question! Our previous top result is now in the second position. Why is that? Because my ‘doc_query_collector’ (which I’ll cover in a future post) recognized that this paragraph appeared in both the lexical and semantic search results, scoring well in each, and promoted it to the top.

Now, switch the query to simply "induction myth," resembling a lexical search. This is now my top result:

Document 1:
Score: 0.9886585235595703
Item Id: ch32
Item Num: 32
Book Title: Unended Quest
Page Number: 163
Chapter Title: 32. Induction; Deduction; Objective Truth
Paragraph Num: 1
Content: There is perhaps a need here for a few words about the myth of
induction, and about some of my arguments against induction. And since at
present the most fashionable forms of the myth connect induction with an
untenable subjectivist philosophy of deduction, I must first say a little more
about the objective theory of deductive inference, and about the objective
theory of truth.

Our previous top result—the one that directly stated, “Thus induction is a myth”—is now in the second position. The current top result is preferred because it contains more occurrences of the words "induction" and "myth."

Notice that with hybrid search, we obtained better results regardless of whether we wanted more lexical or semantic outcomes. This is because we’re truly getting the "best of both worlds" from our search.

In a future post, I’ll explain how I coded the new ‘lex_retriever’ node. There are some interesting aspects to how I implemented it to provide streaming results, ensuring we don’t have to wait on the Large Language Model (LLM). However, I should mention that a lexical search is built into both Haystack and PostgreSQL, so adding a lexical search is not difficult. Here’s the Haystack command to create a ‘keyword retriever’ component:

PgvectorKeywordRetriever(document_store=self._document_store, top_k=self._retriever_top_k)

You can read more about the PgvectorKeywordRetriever here.

Links:

Semantic Search

Hybrid Retrieval

Our Open-Source AI Stack: The Book Search Archive

Tue, 29 Oct 2024 08:55:00 -0600

In our on-going series of AI blog posts, we’ve been developing a simple application that shows off (and tests out) our low cost AI open-source stack. This series of blogs has been developing a way to search and query a book (we’ve been using the Federalist Papers as our example).

We have now released an improved version of this “Book Search” program that we’re calling the “Book Search Archive”. You can find the current code (as it is being developed) in this GitHub repo.

At this point the code is similar to the code we released back in this post but with a number of improvements that we’ll be covering in future posts. The code shows how to develop a simple Retrieval Augmented Generation (RAG) system.

The goal was to develop an AI stack that will allow us to deliver a low-cost AI solution to our clients by using open-source solutions that are still strong enough to be competitive. (As discussed in this post here about a low-cost Tech Support Agent to assist new tech support agents).

I am planning to have future blog posts that will go over changes I’ve made in the Book Search Archive program. But feel free to try this out for yourself in the meantime.

Setting Up the Book Search Archive

The first step is to set up your environment. This post goes over how to setup your environment in detail. There is one difference, however. This new Book Search Archive repo uses more up to date version of code. To catch up to the right versions you need only look at the requirements.txt file in the repo. (Found here). I am planning to eventually upgrade everything, but as I upgrade Python packages I always re-freeze the current requirements into the requirements.txt file. As mentioned in the setup post, you will probably have to install PyTorch separately as pip install doesn’t work right with the PyTorch software for some reason. (Probably because you have to specify where to get the download and inside of a requirements.txt file that isn’t specified).

Features of the Book Search Archive (Current)

The Book Search Archive program is very similar to what we already developed in past posts especially these posts:

Main Articles on Haystack and pgvector:

Other Haystack Related Articles:

Hugging Face Model Related Articles:

The current release includes all functionality from those posts plus the following new features (to be discussed in future posts in detail):

Greatly enhanced meta data collection from EPUB files (which are actually just HTML)
- Example: It now captures page numbers as well as chapter and section titles right out of the text and stores them as meta data. This will allow you to find the text in a paper back book if desired by looking up the quotes used.
A Hybrid Search using Lexical and Semantic search together. Or each one individually if preferred.
Improved text capture. I now capture all text in the book even stuff outside of a paragraph tag.
- Example: Some books I have used have quotes at the top of a page that were getting skipped. Now, nothing gets skipped.
Skipping sections that aren’t helpful
- Okay, actually, I meant to say we don’t skip anything UNLESS we identify it as not helpful. Such as tables of context or bibliographies.
- I have added a number of ways to specify what you do NOT want included in the archive. This improves searching if it doesn’t have to search through things that aren’t helpful to a user.
Improved Document Splitting
- It now splits not only on sentences based on a ‘.’ (as comes default with Haystack if you do a ‘sentence’ search) but also accepts ‘?’ and ‘!’ or even new lines.
- If all of those still fail to allow the Haystack pipeline to avoid truncation then it will fail over to a word split and find a good split based on chunks of words. This assures us that we never get truncated when embedding text.
Loading multiple books at once
- Now you can specify a directory and it will search that directory for all EPUB files (PDFs coming soon!) and loads all of them.
Loading additional books
- No longer do you need to start fresh each time you do a load. Now you can load more books into an existing document store without losing what you already had.
A Custom Document Joiner
- Haystack comes with a Document Joiner that, at least in the version I was running on, screws up text streaming from the LLM. I wrote my down version of a document joiner to avoid this problem.

Expect future posts explaining each of the above at some point.

Future Features

We have plans to include additional features in the Book Search Archive coming soon. These might include:

Streaming from a custom LLM on your own private server
LangChain integration
A full chat interface: have a conversation with the author of the book rather than just get an answer to a single question
Additional architectures (e.g. ReAct and Chain of Thought)
Re-ranker
Improved Semantic Text Chunking
Apache AGE (graph database in PostgreSQL)
Integration with other tools such as: a. LlamaIndex b. LlamaParse to create a graph knowledge base c. NeuralDB (Embedding free indexing) d. DSPy (Prompt Tuning) e. ElasticSearch f. Streamlit (Customizable UI Interfaces)

So, stay tuned for even more great (but low cost and simple) Artificial Intelligence solutions!

An AI Tech Support Agent

Tue, 22 Oct 2024 08:55:00 -0600

It takes months to train a good Tech Support agent, and turnover in this field is notoriously high. Some studies report a staggering 37% turnover rate (Source: The Quality Assurance & Training Connection), while the best figures I've seen still show over 13%, which is alarmingly high. (Source) Support organizations invest in training, only to see agents leave once they become valuable.

So, what can be done?

Imagine having a low-cost, in-house expert that could assist every new tech support agent. With Artificial Intelligence, that's possible—and it won’t break the bank with 'token costs' to OpenAI or Google. Plus, you don't have to worry about sending your private data to these organizations. Because, if needed, we can build you a powerful AI agent using entirely open-source Large Language Models (LLMs).

You Don’t Need the AI to be Customer Facing to get an Immediate ROI

You might also be concerned about allowing customers to interact directly with an AI. What if it does something unexpected?

There’s a lot of potential in the idea of creating an AI Tech Support Agent that could eventually handle the job of human agents. This could cut staffing costs and offer your customers 24-hour tech support at a lower price. And if that’s what you're aiming for, we can certainly help. But what I’m proposing as a first step is something simpler and far more affordable.

I'm talking about an open-source, low-cost AI that every new human agent can interact with—an AI that already knows everything about your products. When a new tech support agent encounters a customer issue, they can chat with the AI, describe the problem, and the AI will quickly pull up the most similar cases from your tech support database. It will then summarize them and suggest solutions for the agent to offer the customer.

With this AI, the cost of training effective human tech support agents drops significantly. Your agents become valuable almost immediately, while your experts are freed up for more complex tasks. Plus, the AI helps train new staff through real-time use. The return on investment is clear.

A Real Life Example of an AI Tech Support Agent

For example, here is an example query used with one of our actual AI agents:

“Vehicle is a 2004 Ford Excursion (2003 engine). The customer is complaining about a rough idle and there are a couple of codes that are set that talk about the high-pressure oil system. They are using an 85200 product and running at economy mode for better fuel consumption.”

After typing that our AI Agent comes back with a summarized list of similar issues:

Case Id: 8767

Customer's Ford has gone through multiple high-pressure oil pumps. Edge product is not likely the cause, as it would set codes if there was an issue with pressure.

Case Id: 42254

Customer experienced rough idle and rough start, with P1233 and P0191 codes indicating fuel pump and fuel line pressure issues. Recommended updating the device and checking the cold air intake.

Case Id: 36396

Customer's truck had a high idle, reaching over 700 RPM. Issue was resolved by reinstalling EGT sensors in the exhaust.

Case Id: 7750

Customer's truck had high and rough idle while running on 100% vegetable oil with an Evolution device. Recommended updating the Evo and checking injector health.

Case Id: 57533

Customer's truck had a high idle for seven months.

Then, using that list, it puts together a punch list of things to have the customer try to resolve their issue:

Update the product to the latest version. (36396, 7750, 57533)
Check for any updates to the product's firmware. (36396)
Check the cold air intake. (42254)
Reinstall the EGT sensors back into the exhaust. (36396)
Check for any other potential hardware failures on the vehicle. (8767)

Notice how the AI references the actual cases so that the human can refer to them or look them up in their support system.

Finally, the AI summarizes its suggestions in a friendly way for the human agent:

“Based on the information provided, it appears that the customer's vehicle may have a problem with the high-pressure oil system. There are two codes set that talk about the high-pressure oil system: P0191 and P1233. These codes indicate that the fuel pump and fuel line pressure are out of range. Similar problems in the past have been resolved by updating the device software (Case Id: 57533). In other cases, the issue was resolved by reinstalling the EGT sensors back in the exhaust (Case Id: 36396). Therefore, I recommend that the customer update the device software and reinstall the EGT sensors back in the exhaust.”

The AI Agent not only has access to your current support database—allowing it to reference similar cases—but it also has access to all of your technical manuals. This means it can instantly search your entire support database as well as your tech manuals for solutions to the user's problem. It can then display or summarize the relevant information, helping your support agents provide faster and more accurate assistance.

For example, here is a real line from a manual that explains how to troubleshoot common problems:

This manual has been added to the AI’s database. When we inform the AI that a user is reporting their camera is frozen, here is the response it provides:

“Based on the user guide, a possible cause of the camera picture freezing is bad connections. To troubleshoot this issue, it is recommended that you cycle the key off and on, and double check the fuse and connectors.”

A Low Cost Open-Source AI Solution

Building an AI Agent like this can be accomplished using primarily open-source models. If needed, we can run the AI locally, ensuring that your private data never leaves your environment. This way, you aren’t sending sensitive information to OpenAI, Google, or any external service—everything stays securely behind your own firewall.

Alternatively, we can train your in-house developers to build something like this themselves. Mindfire TECH offers a wealth of free AI tutorials and sample code on our blog, giving your team the tools and knowledge to get started. This way, you can create and maintain your own AI solutions in-house, with full control over the process.

I recently gave a presentation on our low-cost open-source AI stack to an AI Special Interest Group. You can check out the slide deck here (Link), which provides much more detail on how we developed this affordable AI solution.

Haystack Streaming Text Generation

Tue, 01 Oct 2024 09:00:00 -0600

In this post, we're going to give our sample Retrieval Augmented Generation (RAG) Pipeline a bit of a makeover. Specifically, we’ll tweak it to “stream” results from multiple nodes, letting us show off what we’ve got as soon as it rolls in—no more waiting for everything to pile up at the end.

Right now, our RAG Pipeline has one big flaw: it waits until the entire result is ready before it lets the user see anything. Talk about suspense! Not the best user experience, eh? It's way better if we start showing results the moment we have something. Not only does it feel snappier, but the user also gets the illusion of speed. Magic!

As it stands, our pipeline first grabs all the documents that match the query, then the Large Language Model (LLM) uses them to craft a response. The hitch? Haystack pipelines are synchronous, so we don’t get a peek at the retrieved documents until the LLM finishes its grand finale. But for document retrieval, the user might actually want to start digging into the docs while the LLM is still generating its response. Multitasking at its finest!

Ideally, we want the node fetching the documents to hand them over right away before passing them off to the LLM. And wouldn’t it be great if the LLM could stream its response a few words at a time, so we can watch it unfold in real-time?

The good news is, we can make this happen using a combo of custom Haystack components and Haystack’s streaming callbacks. The catch? It’s a bit more complicated. I have it on good authority (from a Deepset employee on Haystack’s own discord) that Haystack might add asynchronous pipelines in the future, but for now, custom components and streaming callbacks are the best trick in the toolbox.

Haystack’s Streaming Callback

To make streaming work correctly with our RAG pipeline we’ll need to make changes to all three of our python files. As a reminder, in our last post, we broke our sample code into three files:

The Generator Models: generator_model.py (Latest version: may include future posts)
The Document Processor: document_processor.py (Latest version)
The RAG Pipeline: rag_pipeline.py (Latest version)

The first change we need to make is to the Generator Models (i.e. generator_model.py). We need to enable them to accept a streaming callback function that Haystack can use. Fortunately, Haystack’s Hugging Face components—whether you’re using the local model or the API—already support passing in a streaming callback function. That’s what we’ll be using under the hood.

However, we want our pipeline to handle this dynamically, regardless of whether the specific Haystack Generator in use supports streaming callbacks. If the generator doesn’t support them, it should simply default to synchronous behavior and return the result at the end of the pipeline.

To make this kind of flexible behavior possible, we’ll adjust our generator classes so that they always pass a default streaming callback function for Hugging Face models. If the user doesn’t request streaming, the callback will just be ignored. Here’s how we can modify the Abstract Hugging Face Generator class (which underlies both the local and API based generators for Hugging Face models) to achieve that:

class HuggingFaceModel(GeneratorModel, ABC):
    def __init__(self,
                 model_name: str = 'google/gemma-1.1-2b-it',
                 max_new_tokens: int = 500,
                 temperature: float = 0.6,
                 password: Optional[str] = None,
                 streaming_callback: Optional[Callable[[StreamingChunk], None]] = None,
                 verbose: bool = False) -> None:
        super().__init__(verbose)

        self._max_new_tokens: int = max_new_tokens
        self._temperature: float = temperature
        self._model_name: str = model_name
        self._streaming_callback: Optional[Callable[[StreamingChunk], None]] = streaming_callback

        if password is not None:
            hf_hub.login(password, add_to_git_credential=False)

    @property
    def streaming_callback(self) -> Optional[Callable[[StreamingChunk], None]]:
        return self._streaming_callback

    @streaming_callback.setter
    def streaming_callback(self, value: callable(StreamingChunk)) -> None:
        self._streaming_callback = value

    def generate(self, prompt: str) -> str:
        return self._model.run(prompt)

    def _default_streaming_callback_func(self, chunk: StreamingChunk):
        # This is a callback function that is used to stream the output of the generator.
        # If you are not using a streaming generator, you can ignore this method.
        if self._streaming_callback is not None:
            self._streaming_callback(chunk)

So, we added a parameter for a streaming callback in the initialization. Unfortunately, this isn’t going to be useful the way we currently have our RAG pipeline class. What we really need is a way to pass in the model without specifying a streaming callback and then let the RAG Pipeline add one later. Hugging Face does not support such an ability, so we’re going to cheat. We’re going to always give the Hugging Face component our “default_streaming_callback_func”. This function checks to see if we’ve given the class an actual streaming callback function. If we haven’t, it does nothing. If we have, it calls it.

This approach magically allows us to now create a property four all Hugging Face classes that we can just assign a streaming callback to. In fact, we can change which function is the streaming callback function on the fly now! Pretty neat, eh?

There are also some small changes to our HuggingFaceAPIModel and HuggingFaceLocalModel classes to allow them to pass in the streaming callback function to their initialization function as well. I won’t go over these and honestly, they aren’t needed for our purposes.

The upshot of these changes is that all Hugging Face models we create (if we do so via our generator model classes) will now have a way to specify a streaming callback. The RAG pipeline will utilize this new feature if we tell it to.

I did not add streaming callback functionality to our Google Gemini model class wrapper because the version of Haystack I’m on does not support such functionality. But Google Gemini is so blazing fast it won’t really matter that much. But that explains why I only added the streaming callback functionality to our Hugging Face models class wrapper.

I made some adjustments to the Document Processor file as well, though to be honest those were more code hygiene issues I’ve been meaning to get to. So, I’ll not go over those changes either as they aren’t truly relevant to our current purposes.

Building a “Streaming” Haystack Component

Haystack natively supports streaming callbacks for their Hugging Face component models. But what about the document retriever component? Haystack does not have any sort of ‘streaming callback’ function on those components. But what if we want to show off the retrieved documents while we want for the model to generator its own response?

One way to handle that is to simply write our own document retrieving component that wraps the Haystack PgvectorEmbeddingRetriever component but streams out the documents via print statements to the console before returning the documents to the next node in our pipeline. Here is what I built:

@component
class StreamingRetriever:
    def __init__(self, retriever: PgvectorEmbeddingRetriever):
        self.retriever = retriever

    @component.output_types(documents=List[Document])
    def run(self, query_embedding: List[float]) -> Dict[str, Any]:
        # Create a dictionary for the expected format if necessary
        documents = self.retriever.run(query_embedding=query_embedding)['documents']
        print_documents(documents)
        # Return a dictionary with documents
        return {"documents": documents}

The printdocuments command is taken straight from our old generateresponse method, so it may look familiar:

def print_documents(documents: List[Document]) -> None:
    for i, doc in enumerate(documents, 1):
        print(f"Document {i}:")
        print(f"Score: {doc.score}")
        if hasattr(doc, 'meta') and doc.meta:
            if 'title' in doc.meta:
                print(f"Title: {doc.meta['title']}")
            if 'section_num' in doc.meta:
                print(f"Section: {doc.meta['section_num']}")
        print(f"Content: {doc.content}")
        print("-" * 50)

We then need to adjust our pipeline to use this new component – but we only want to do that IF the user asks for streaming. Here is the adjusted code to build our RAG pipeline:

    def _create_rag_pipeline(self) -> None:
        self._setup_embedder()
        self._setup_generator()
        prompt_builder: PromptBuilder = PromptBuilder(template=self._prompt_template)

        rag_pipeline: Pipeline = Pipeline()

        # Add the query embedder and the prompt builder
        rag_pipeline.add_component("query_embedder", self._sentence_embedder)
        rag_pipeline.add_component("prompt_builder", prompt_builder)

        # If streaming is enabled, use the StreamingRetriever
        if self._can_stream():
            streaming_retriever: StreamingRetriever = StreamingRetriever(
                retriever=PgvectorEmbeddingRetriever(document_store=self._document_store, top_k=5))
            rag_pipeline.add_component("retriever", streaming_retriever)
        else:
            # Use the standard retriever if not streaming
            rag_pipeline.add_component("retriever",
                                       PgvectorEmbeddingRetriever(document_store=self._document_store, top_k=5))

        # Add the LLM component
        if isinstance(self._generator_model, gen.GeneratorModel):
            rag_pipeline.add_component("llm", self._generator_model.generator_component)
        else:
            rag_pipeline.add_component("llm", self._generator_model)

        if not self._can_stream():
            # Add the merger only when streaming is disabled
            rag_pipeline.add_component("merger", MergeResults())
            rag_pipeline.connect("retriever.documents", "merger.documents")
            rag_pipeline.connect("llm.replies", "merger.replies")

        # Connect the components for both streaming and non-streaming scenarios
        rag_pipeline.connect("query_embedder.embedding", "retriever.query_embedding")
        rag_pipeline.connect("retriever.documents", "prompt_builder.documents")
        rag_pipeline.connect("prompt_builder", "llm")

        # Set the pipeline instance
        self._rag_pipeline = rag_pipeline

Note the various places where I check if we’re asking for streaming or not via the “_can_stream()” method. If we are streaming, we use the StreamingRetriever in place of the PgvectorEmbeddingRetriever. And if we are not streaming, we add back on the final custom merger node to bring the documents and the LLM generated results together for output.

Here is the code for canstream():

    def _can_stream(self) -> bool:
        return (self._use_streaming
                and self._generator_model is not None
                and isinstance(self._generator_model, gen.GeneratorModel)
                and hasattr(self._generator_model, 'streaming_callback'))

This probably isn’t the best way to do this. For one thing, it turns off streaming the documents out to the user for no reason other than that there is no LLM streaming callback available. But this does keep things simple for out toy example. Basically, I setup the streaming version of the pipeline but only if the following criteria are matched:

The user specified they want it via the use_streaming parameter
We actually have a generator model
The generator model is of type GeneratorModel (i.e. we’re not passing in a direct Haystack generator. In the past we supported that, but currently the code doesn’t work with anything but our GeneratorModel class or subclasses of it.)
The model has an attribute of ‘streaming_callback’.

I also changed the generate_response method to NOT show the final response if it is to be streamed. I’ll not show off that code here as it isn’t that interesting.

Results

The end results are exactly what I hoped for! The documents display to the user as soon as they are retrieved from the PostgreSQL datastore and then we can watch the LLM generate its response one word at a time. This is a much-improved user experience!

Avoiding Text Truncations in RAG

Wed, 25 Sep 2024 09:00:00 -0600

In past blog posts, we’ve been building a cost-efficient Retrieval Augmented Generation (RAG) pipeline using open-source tools such as PostgreSQL database, pgvector, Hugging Face open-sourced models, and Haystack from Deepset.AI to build the pipelines. You can find the post on environment setup here and the posts on building the pipeline code here and here. Most recently we’ve added to the code base an ability to use Google Gemini as the Large Language Model (LLM) and even using the Hugging Face API to host an open-source LLM.

One thing I’ve been bothered about is that the sentence transformer/embedder model I’ve been using is the default model used by Haystack which is sentence-transformers/all-mpnet-base-v2. This model isn’t the best at embedding sentences and its context window (explained below) isn’t very large. The danger is that we’ll break up the documents sent to the database into chunks too large for the model and it will end up truncating the text and it won’t be part of the vector saved. This could degrade the performance of our document retrieval system.

In this post we’re going to make sure we’ve embedded our documents such that the text isn’t truncated. We’ll also try out models, such as Alibaba-NLP/gte-large-en-v1.5, that has a larger context window.

A Note on the Updated Code Base

Up to this point I’ve tried to keep all my code in one file. But that is starting to get unwieldly so this is the last post I’m going to offer a single file for my code, which for this post is found here. Instead, I’ve now split the code into three files:

The Generator Models: generator_model.py (Latest version: may include future posts)
- This file contains a virtual class called GeneratorModel that allows you to wrap any LLM model up for the RAG pipeline with certain default properties and methods that it can rely on existing.
- It also contains a concrete classes such as HuggingFaceLocalModel, HuggingFaceAPIModel, and GoogleGeminiModel. So that you can switch to whichever model you wish to use.
The Document Processor: document_processor.py (Latest version)
- This file contains the class DocumentProcessor which contains the Haystack pipeline to build the document store from an EPUB file.
The RAG Pipeline: rag_pipeline.py (Latest version)
- This file contains the class RagPipeline which contains the actual RAG pipeline used to query the document store.

Most of this code is pretty similar to what we’ve previously developed but now broken up into separate files to make things more manageable. So I’m not going to go over all the code in detail. Instead, I’ll just go over the specifically relevant code for how to make the document processor intelligently break up the EPUB documents.

Haystack Support for SentenceTransformer

When we previously (in past posts) loaded the Hugging Face ecosystem that included the SentenceTranformer module. The SentenceTransformer module is actually from sbert.net. (API Documentation found here). As the sbert.net website explains:

“Sentence Transformers (a.k.a. SBERT) is the go-to Python module for accessing, using, and training state-of-the-art text and image embedding models. It can be used to compute embeddings using Sentence Transformer models or to calculate similarity scores using Cross-Encoder models. This unlocks a wide range of applications, including semantic search, semantic textual similarity, and paraphrase mining.”

Haystack supports SentenceTransformer indirectly through two important classes:

SentenceTransformersDocumentEmbedder: A component for embedding text in a Haystack Document class instance. We’ll use this to creating embedding vectors for our document store. (API Documentation found here).
SentenceTransformersTextEmbedder: A component for embedding text in a string. This is used for embedding queries in the RAG pipeline. (API Documentation found here).

Here is the general Haystack documentation on sentence embedders.

Just What is the Context Length for sentence-transformers/all-mpnet-base-v2?

In previous posts, I had some code to figure out the context window length for the sentence embedder we’re using. Turns out, that code was incorrect, and we’ll need to fix that.

But first, what is the ‘context length’ for a sentence transformer/embedder, you ask? Well, all language models have a context length, which is basically how much text (measured in tokens) the model can process at one time. Think of it like when you’re chatting with ChatGPT—it’s great at keeping track of the recent conversation, but as texts scrolls farther up the screen, it starts forgetting. That’s because there’s a limited window of text the model can "see," and anything outside of that gets lost in the void.

So, what’s the context window for our sentence embedder that generates the vectors we’re saving to the PostgreSQL database? In the past, I used something like the code below to figure it out:

    @staticmethod
    def _get_context_length(model_name: str) -> Optional[int]:
        config: AutoConfig = AutoConfig.from_pretrained(model_name)
        context_length: Optional[int] = getattr(config, 'max_position_embeddings', None)
        if context_length is None:
            context_length = getattr(config, 'n_positions', None)
        if context_length is None:
            context_length = getattr(config, 'max_sequence_length', None)
        return context_length

This uses Hugging Faces AutoConfig component to grab the properties (contained in the config.json file) for a model. When I checked what the context length was for the default sentence embedder used by Haystack (i.e. sentence-transformers/all-mpnet-base-v2) it came back as 514 tokens. This means (or is supposed to mean) it can handle up to 514 tokens before it starts truncating text. Just to be thorough, I double-checked. AutoConfig pulls these numbers straight from the model’s config.json file, which confirms that 514 is indeed the maxembeddingpositions (i.e. this models property for the context length).

Now, up until recently, I figured this was perfectly fine. After all, 514 tokens can cover a decent chunk of text, so I wasn’t too concerned about cutting anything off when embedding paragraphs. But just to be extra careful, I set up the document processing pipeline with a DocumentSplitter node, instructing it to split text after 10 sentences. I thought, “This should work like a charm.”

Naturally, I was wrong.

Lesson learned: always test these things, especially when you’re building a production-ready app.

There is a more direct way to get the context length for the sentence embedder. Here is my revised code:

    @property
    def context_length(self) -> Optional[int]:
        self._setup_embedder()
        if self._sentence_embedder is not None and self._sentence_embedder.embedding_backend is not None:
            return self._sentence_embedder.embedding_backend.model.get_max_seq_length()
        else:
            return None

Here I’m not bothering with the config.json (as read by AutoConfig) and I’m instead grabbing the backend model directly and using the ‘getmaxseqlength()’ method to directly find out what the real context length is. And it turns out that sentence-transformers/all-mpnet-base-v2 actually only has a contextlength of 384.

That’s short enough that I’m probably regularly truncating my document fragments as I embed them for my document store.

Turns out (according to this post) that you can’t trust the config.json file. (Compare to here, here, and here).

Avoiding Truncation

So now that we know we only have 384 tokens for each document fragment, how can we guarantee that every single document fragment stored in the PostgreSQL will fit within that context window?

Moreover, it would be nice if we could come up with a solution to this problem that could dynamically resize the size of the document fragments if we later switch to a sentence embedder with a larger context window (which we’re going to do in this post!)

There is a larger issue here that I’m not really addressing. Ideally, how should we break up a document (such as an EPUB or PDF)? By page? By paragraph? By sentence? Multiple sentences?

What we want is for the document fragments we store in PostgreSQL to be semantically ‘self-contained’ as much as possible. We want each document fragment to carry a single thought. That sounds like a paragraph to me, though with a large enough context length it might make sense to try story by page instead. (But, so below for an argument against embedding a full page).

Luckily, we are using my EPUB code to load EPUB files which allows us to automatically break up the document into paragraphs. And most paragraphs probably fit even into the 384 token context window \– though a few paragraphs are definitely too long for that window. In such cases, we really want to break up the paragraph by sentences until it fits within the context window so that we don’t truncate any of the text. But we’d want to be as close to a paragraph as possible. (Note: this is assuming we decided that for out we want to semantically break documents into paragraphs. Your own app may have different needs).

Strategically Breaking Paragraphs to Avoid Truncation

Our proposed strategy to avoid truncation is as follows:

First try embedding the entire paragraph. Does the number of tokens exceed the context length (for the default model that is 384 tokens)? If it fits, then the paragraph will be saved embedded and saved as a vector to our document store.
If the paragraph is too large, we’ll try to grab 10 sentences at a time (with an overlap of 1 sentence). We’ll then check the number of tokens required. If it fits, we’ll save it to the document store.
If it still doesn’t fit then we’ll reduce to 9 sentences and try again. If it still doesn’t fit, we’ll drop an additional sentence, etc.

Note that this means we’re going to have to check the actual token length using the sentence transformer model which will slow down our pipeline quite a bit. But the document loading pipeline (for our toy app anyhow) is only loaded once, so this isn’t a big deal for us.

A Custom Component To Avoid Truncation

I don’t really want to write my own code for DocumentSplitter, but the built-in DocumentSplitter doesn’t check if the embedding truncated text or not. So, we’re going to write a custom component to do the job but it will still – under the hood – use Haystack’s DocumentSplitter. Here is my proposed code:

    @component
    class _CustomDocumentSplitter:
        def __init__(self, embedder: SentenceTransformersDocumentEmbedder):
            self.embedder: SentenceTransformersDocumentEmbedder = embedder
            self.model: SentenceTransformer = embedder.embedding_backend.model
            self.tokenizer = self.model.tokenizer
            self.max_seq_length = self.model.get_max_seq_length()

        @component.output_types(documents=List[Document])
        def run(self, documents: List[Document]) -> dict:
            processed_docs = []
            for doc in documents:
                processed_docs.extend(self.process_document(doc))

            print(f"Processed {len(documents)} documents into {len(processed_docs)} documents")
            return {"documents": processed_docs}

        def process_document(self, document: Document) -> List[Document]:
            token_count = self.count_tokens(document.content)

            if token_count <= self.max_seq_length:
                # Document fits within max sequence length, no need to split
                return [document]

            # Document exceeds max sequence length, find optimal split_length
            split_docs = self.find_optimal_split(document)
            return split_docs

        def find_optimal_split(self, document: Document) -> List[Document]:
            split_length = 10  # Start with 10 sentences
            while split_length > 0:
                splitter = DocumentSplitter(
                    split_by="sentence",
                    split_length=split_length,
                    split_overlap=min(1, split_length - 1),
                    split_threshold=min(3, split_length)
                )
                split_docs = splitter.run(documents=[document])["documents"]

                # Check if all split documents fit within max_seq_length
                if all(self.count_tokens(doc.content) <= self.max_seq_length for doc in split_docs):
                    return split_docs

                # If not, reduce split_length and try again
                split_length -= 1

            # If we get here, even single sentences exceed max_seq_length
            # So just let the splitter truncate the document
            # But give warning that document was truncated
            print(f"Document was truncated to fit within max sequence length of {self.max_seq_length}: "
                  f"Actual length: {self.count_tokens(document.content)}")
            print(f"Problem Document: {document.content}")
            return [document]

        def count_tokens(self, text: str) -> int:
            return len(self.tokenizer.encode(text))

This code does exactly what we discussed above. It first tries to embed a paragraph but verifies there was no truncation. If that doesn’t work, it tries 10 sentences and tries again. It keeps dropping the number of sentences until it fits within the context length of the sentence embedder. It will also automatically adjust to whatever the actual context length is for whatever model we are using. So, if we want to swap out the default sentence transformer model for a better one, this code will adapt.

You might notice that my code allows for an overlap of 1 sentence. This means that when I split up the text, I allow one sentence to overlap between fragments. This helps ensure we don’t lose too much context when we split up the text of a paragraph into document fragments.

This is still a fairly primitive way to split up documents. There are more advanced techniques available that we’ll cover in future posts. But for now, this should work pretty well. And in any case, this technique is a useful tool in our toolbox for when we use more advanced techniques.

Using an Improved Sentence Embedding Model

For the Federalist Papers EPUB we’re using, there are 1482 paragraphs. So if every single one fits within the context length we should have 1482 ‘documents’ (or rather document fragments). Using the default model (sentence-transformers/all-mpnet-base-v2), here is the result that is reported when we use our custom document splitter:

Processed 1482 documents into 1597 documents

That’s not too bad, but let’s see if we can use a better model that will allow every single paragraph to be stored in our documents database.

There is a Hugging Face Leaderboard for sentence transformers found here. Most of the really good sentence transformers/embedders are much too large for my laptop, but I do see there are some smaller ones available. I selected Alibaba-NLP/gte-large-en-v1.5 which is quite a ways down on the list, but via testing I confirmed it works out of the box with my environment (some others do not!) I like the fact that it has a context window of 8192. That is huge compared to the 384 we’ve been playing with. (Note also the new size of the embedding vector is 1024).

I first switch my code to use this new model. Something like this will work:

model: GeneratorModel = HuggingFaceAPIModel(password=hf_secret, model_name="HuggingFaceH4/zephyr-7b-alpha")
rag_processor: HaystackPgvector = HaystackPgvector(table_name="federalist_papers",
                                                   recreate_table=False,
                                                   book_file_path=epub_file_path,
                                                   generator_model=model,
                                                   embedder_model_name="Alibaba-NLP/gte-large-en-v1.5")

A note of caution: you must use the same sentence transformer/embedder model in both the document pipeline and the rag pipeline or else you’ll get back results.

Since we coded this to check for the size of the context window and the size of the embedding vector dynamically, our code will automatically adjust to this improved model.

With this new model, I get the following result:

Processed 1482 documents into 1482 documents

Perfect! Now every single paragraph is its own document fragment! We have successfully embedded our document into fragments that all fit into the model’s context window.

What Is the Optimal Number of Tokens to Embed?

A good question is ‘what is the optimal number of tokens to embed?’ In this post we’re sticking with the simple assumption that a paragraph is a semantically interesting unit and will be close to appropriately sized. We also talked about the default sentence embedder which has 384 tokens and a replacement embedder that allowed 8192 tokens by comparison! But does it even make sense to embed such a huge chunk of text? A context window of 8192 is huge and would significantly dilute any sort of search via cosine similarity.

It turns out there was a recent paper (Wang et al., 2024) where they empirically found the optimal number of tokens and that study found that between 256 and 512 tokens was generally the right size to avoid diluting your search. Surprise! The default sentence transformer’s 384 tokens are right smack in the middle of that. I doubt that is a coincidence. The default sentence transformer was undoubtedly picked based on working well in real life. By comparison, our improved model with a context window of 8192 is way out of whack. (Though we are not utilizing such a large window due to sticking with paragraphs). This is why you should probably not look at embedding full pages of text.