A Local OCR Plugin for FiftyOne Datasets

A community plugin for FiftyOne adds optical character recognition to image datasets by running PyTesseract, the Python bindings for the Tesseract OCR engine. The plugin processes documents directly inside FiftyOne and converts the recognized text into labels stored on the dataset, so text extraction happens alongside the rest of a computer vision workflow rather than in a separate tool. The project is published at https://github.com/jacobmarks/pytesseract-ocr-plugin and is built by Jacob Marks.

What the Plugin Does

The plugin exposes a single operator named run_ocr_engine. According to the project README, the operator runs the PyTesseract OCR engine on the documents in a dataset, converts the results to FiftyOne labels, and stores both individual word predictions and block-level predictions on the dataset. That means a single run produces two levels of detail: fine-grained word detections and coarser blocks of text, both attached to the samples they came from.

An update noted in the README, dated 2023-10-19, added support for customizing the prediction fields and an embedded field for OCR text. This gives some control over where results are written when the operator runs.

Because the plugin relies on Tesseract running through PyTesseract, the recognition work happens on local infrastructure. The README does not describe any external API calls, so the OCR step stays within the same environment that holds the dataset.

Running the Operator

Installation uses FiftyOne’s plugin download command, followed by installing the plugin’s Python requirements:

fiftyone plugins download https://github.com/jacobmarks/pytesseract-ocr-plugin
pip install -r requirements.txt

Once installed, the operator is reached from the FiftyOne App. The README describes two ways to open it: through the App’s action menu, or by pressing the backtick (`) key and selecting the operator from the dropdown menu that appears.

The plugin also adapts to what is currently in view. If a view is loaded or samples are selected, the operator offers the choice of running the OCR engine on only those samples or on the entire dataset. Runs can happen in the foreground, or the execution can be delegated to a background job, which is useful when a dataset is large enough that processing should not block the App.

Searching the Results

The README points to a follow-on step for working with the extracted text. After OCR predictions are generated, they can be searched using the separate Keyword Search plugin, also from the same author. This pairs the text that PyTesseract recognizes with a way to query it, so a dataset can be filtered down to samples that contain particular words once the labels exist.

A Note on Scope

The plugin documents only the PyTesseract and Tesseract engine. Claims about a single release bundling multiple OCR backends, specific throughput numbers, GPU acceleration, distributed processing, or particular organizations using it are not part of this project’s documentation and should not be assumed. What the source supports is a focused tool: it takes images in a FiftyOne dataset, runs Tesseract over them locally, and writes word-level and block-level text predictions back as labels that the rest of the FiftyOne workflow can use.