Can a Raspberry Pi turn spoken reflections into useful notes without sending audio to the cloud? In this walkthrough, you will build Voice Notes, a single-user voice notebook that captures speech in the browser, transcribes it with Microsoft Foundry Local, and drafts a summary on-device. Along the way, educators and learners can explore audio processing, web APIs, model lifecycle management, and responsible AI on resource-constrained hardware.

Video demo: Watch the Voice Notes demo on Bilibili.

Scope and versions: This is an educational prototype based on a recorded Raspberry Pi 5 experiment, not a managed classroom service or a Raspberry Pi hardware-certification statement. The walkthrough uses Microsoft Foundry Local Python SDK 2.1.0 and Foundry Local CLI Preview 0.10.3. These are separate packages with independent version numbers. Commands, model aliases, and APIs can change. The community sample is pinned to a specific revision so readers can follow the same implementation.

What you'll build

The finished application provides three workflows:

• Live transcription: Capture speech in the browser and display recognized text while recording.
• File transcription: Upload up to ten WAV or MP3 files and process them sequentially with Whisper.
• Local note generation: After a live recording ends, unload the speech model and use Qwen3 to extract topics, key points, decisions, and action items.

Completed live sessions provide a playable WAV recording, a TXT transcript, and a JSON record. File-upload jobs provide TXT and JSON results; they do not create a new recording or automatically generate a summary.

The sample's interface and generated summaries are in English. Transcripts retain the language recognized by the selected speech model.

Local inference does not require an Azure subscription, an Azure resource, or a cloud inference API key. Installation, model acquisition, and any required runtime-component downloads need connectivity. After preparation, you will explicitly test the cached workflow without internet access rather than assuming it is offline-ready.

Prerequisites

Hardware and operating system

 

These are reproduction recommendations, not official minimum specifications. 

Figure 1. The source experiment's Raspberry Pi 5 and camera-based audio-input hardware. For this walkthrough, connect a working audio input and run the browser on the Pi itself.

Software, accounts, and models

You will need:

• Python 3.11 or later with an ARM64 wheel available for the selected SDK version.
• Git, curl, tar, and Python virtual-environment support.
• The sample's Python dependencies: Flask and foundry-local-sdk on Linux.
• A modern browser supporting getUserMedia, Web Audio, and AudioWorklet.
• Internet access for the initial downloads.
• A short speech recording you own or have permission to use.

A GitHub account is not required to clone the public sample. The sample offers these model aliases:

 

The official live-transcription guide also shows the multilingual alias nvidia-nemotron-3.5-asr-streaming-multilingual-0.6b. Do not assume the two names are interchangeable in every catalog. The sample validates a fixed list of allowed aliases, so a catalog-name change must be reflected in the backend constants, not just typed into an API request.

Architecture

Foundry Local is an on-device runtime and SDK, not simply a command-line server. In this application, Flask calls Foundry Local in the same Python process. There is no separate Foundry REST endpoint in the application's inference path.

Figure 2. Capture, web application, inference, and storage all run on the Raspberry Pi. Model acquisition needs connectivity; prepared inference uses local resources.

The application runs one inference job at a time. This follows Foundry Local's single-user, hardware-constrained design and avoids keeping speech and chat models in memory together. For live audio, the browser sends approximately 100 ms of sound per chunk. Flask accepts ordered chunks, saves the recording, and feeds the native speech session. The UI polls for results, waiting 100 ms after each completed status request rather than issuing overlapping requests. Neither 100 ms chunks nor 100 ms polling means 100 ms recognition latency. Model processing and finalization are separate costs. The browser captures the Pi's microphone and sends audio only to the loopback-bound application on the same device. The inference path does not call a cloud model service.

Step-by-step walkthrough

All commands in this walkthrough run in a Bash terminal on the Raspberry Pi. Open the web application in a browser running on that same Pi.

1. Check the Pi and prepare the Python environment

Confirm the architecture and available resources:

uname -m
cat /etc/os-release
free -h
df -h /

Expect aarch64 for the ARM64 environment used here. armv7l indicates a 32-bit installation and cannot use the ARM64 archive below.

Install the basic tools if necessary:

sudo apt update
sudo apt install -y git curl ca-certificates tar python3 python3-venv

Clone the sample, select the article revision, and create an isolated environment:

cd "$HOME"
git clone https://github.com/shijiong/flask-audio-transcription.git
cd flask-audio-transcription
git checkout --detach e3047faa6e9181332e830b7f435b0ee6109521ef

python3 -m venv .venv
source .venv/bin/activate
python -m pip install --upgrade pip
python -m pip install -r requirements.txt "foundry-local-sdk==2.1.0"

The repository allows a wider SDK 2.x dependency range. The explicit constraint above selects the version used in the source experiment; it does not claim that 2.1.0 is the latest version. Evaluate upgrades separately.

On Linux, use foundry-local-sdk, not the Windows-specific foundry-local-sdk-winml package. Both expose the Python import name foundry_local_sdk.

If pip cannot find a compatible distribution, check the operating-system architecture and Python version against the package's available wheels. Do not solve that error by installing an unrelated package or renaming the import.

2. Optionally install the CLI to validate file transcription

The application does not require the CLI. This separate exercise is useful for checking the device, catalog, and a speech model before working through the web application.

The CLI Preview 0.10.3 release includes a Linux ARM64 archive:

cd "$HOME/Downloads"
curl -fL -o foundry-0.10.3-linux-arm64.tar.gz https://github.com/microsoft/foundry-local/releases/download/cli-preview-0.10.3/foundry-0.10.3-linux-arm64.tar.gz

printf '%s %s\n' 'c530e6354f29335fee4fd676b0f945e53a2dedcfc73c64fe336a39d27dbaec82' 'foundry-0.10.3-linux-arm64.tar.gz' | sha256sum --check

Continue only if checksum verification reports OK. The digest above applies to this exact release asset.

The screenshots in this step come from the original CLI 0.10.3 session. Some commands were run from its lib directory and the original audio filenames differ from the exercise filenames below. The commands in this article consistently use ./lib/foundry from the installation root.

tar xzf foundry-0.10.3-linux-arm64.tar.gz -C "$HOME"
cd "$HOME/foundry-0.10.3-linux-arm64"

Keep the complete extracted directory; copying only the executable can omit required runtime files.

Figure 3a. Extract the Linux ARM64 package into the user's home directory.Figure 3b. The complete installation directory is retained on the Pi.

Check the executable and inspect the device:

./lib/foundry --version

Figure 4. Confirm the CLI version before following version-specific commands.

./lib/foundry status

Figure 5. The recorded status output identifies the device and resources.

Inspect the command help and model catalog:

./lib/foundry --help
./lib/foundry model --help

Figure 6. Use the installed CLI's help

./lib/foundry model list
./lib/foundry server status

Figure 7. The recorded catalog includes CPU model variants.Figure 8. The CLI server becomes ready.

Inspect and cache the lightweight file-transcription model:

./lib/foundry model info whisper-tiny
./lib/foundry model download whisper-tiny

Figure 9a. The recorded Whisper Tiny download reports approximately 131 MB.Figure 9b. Download completion validates acquisition

./lib/foundry transcribe --help

Figure 10. Check the-m,-l, and-foptions before processing an audio file.

For an existing English speech file saved as Documents/Recording.mp3, run:

./lib/foundry transcribe -m whisper-tiny -l en -f "$HOME/Documents/Recording.mp3"

Replace the path with your actual recording. The screenshot below shows the original English MP3 test rather than the synthetic classroom reflection introduced later.

Figure 11. The original CLI experiment produces an English transcript.

For an existing Chinese recording, use a Chinese language hint:

./lib/foundry transcribe -m whisper-tiny -l zh -f "$HOME/Documents/output_16k_mono.wav"

Figure 12. The original Chinese WAV test confirms that the CLI can produce Chinese text.

These file-based CLI checks are separate from the timed live-streaming benchmarks discussed in Test the solution. After this exercise, stop the CLI server so it does not compete with the web application for memory:

./lib/foundry server stop

3. Prepare the models through the application's SDK path

Return to the sample and activate the same environment:

cd "$HOME/flask-audio-transcription"
source .venv/bin/activate

Save the following as prepare_models.py in the repository root:

from src.foundry_backend import FoundryBackend

backend = FoundryBackend()
aliases = (
    "nemotron-speech-streaming-en-0.6b",
    "whisper-tiny",
    "qwen3-0.6b",
)

for alias in aliases:
    with backend.model(alias, print) as model:
        print(f"Ready: {alias} -> {model.info.id}")

Run it before starting Flask, not alongside another inference process:

python prepare_models.py

FoundryBackend is the sample's adapter, not a class supplied by the SDK. It initializes FoundryLocalManager, registers execution providers, looks up each alias through manager.catalog.get_model(), downloads the model, loads it, and unloads it when the context ends.

A missing model causes an explicit error. The application does not silently substitute another model. Add the multilingual model or whisper-small to this preparation list only if you intend to use them.

The adapter also contains a Linux SDK 2.1.0-specific compatibility workaround, setting CI=1 before SDK initialization for the issue described in the sample README. Treat this as version-scoped sample behavior, not a universal telemetry configuration or an offline guarantee.

4. Configure storage and start the workbench

Use the default local data directory, or choose another writable location:

export TRANSCRIPTION_DATA_DIR="$HOME/voice-notes-data"
export PORT=5000
python src/app.py

The sample binds to 127.0.0.1:5000 with debug mode disabled. Run a single application process: the active job, inference slot, and native sessions are held in memory. Multiple worker processes or the Flask debug reloader can create conflicting state and duplicate model allocations.

On the Pi, open:

http://127.0.0.1:5000

Browser microphone APIs require a secure context. The Pi's loopback address qualifies. Keep the application bound to loopback and use the browser and microphone on the Pi for this walkthrough.

5. Capture speech and generate notes

1. Select Live transcription.
2. Choose a microphone and nemotron-speech-streaming-en-0.6b.
3. Select English and click Start live transcription.
4. Grant microphone permission, wait for the model to become ready, and check that the input-level indicator responds.
5. Speak a short reflection.
6. Click Stop & summarize, then wait for finalization and summary generation.

Behind the interface, the browser converts its microphone stream to 16 kHz, mono, signed 16-bit little-endian PCM. A full 100 ms chunk contains 1,600 samples, or 3,200 bytes. The last chunk may be shorter.

Each upload carries an increasing X-Audio-Sequence number. Retrying an accepted sequence does not append duplicate audio. On stop, the browser flushes its remaining samples, uploads queued audio, and finishes the input stream before the native session closes.

The UI understands interim and confirmed text, but a model is not guaranteed to emit interim results. Interim hypotheses replace earlier interim text; confirmed segments are appended; the final aggregate replaces the accumulated transcript rather than duplicating it.

After recognition finishes, the service unloads the speech model and loads Qwen3. If summarization fails, the warning remains visible and the application still attempts to preserve the recording and transcript.

Figure 13. The live-transcription tab before recording. Choose the Pi's microphone, model, and language, then start a session. This existing screenshot shows an idle interface, not a completed benchmark result.

6. Transcribe existing audio files

Select Audio files, choose whisper-tiny, select a language, and upload one or more WAV or MP3 files. The sample permits up to ten files and a 100 MiB total request limit. Files are processed sequentially. A decoding or inference failure for one accepted file does not discard successful results from the others. For a predictable first test, use 16 kHz mono audio and 16-bit PCM encoding for WAV. If FFmpeg is installed, an optional preparation command is:

ffmpeg -i input.mp3 -ar 16000 -ac 1 -c:a pcm_s16le lesson.wav

The web application does not run an external conversion step, and its extension checks do not validate every audio-format property. The published benchmark also demonstrates that the runtime accepted a 24 kHz stereo MP3. Therefore, 16 kHz mono is preparation guidance here, not a demonstrated universal file-input restriction.

Figure 14. The file-transcription tab before upload. The shared summary panel describes the live workflow; uploading files does not automatically generate summaries.

Code and configuration

A compact file-to-summary pipeline

To see model sequencing without the browser, save this as transcribe_and_summarize.py in the repository root, next to a short Recording.mp3 file:

from pathlib import Path

from src.foundry_backend import FoundryBackend

audio_path = Path("Recording.mp3")
if not audio_path.is_file():
    raise FileNotFoundError(f"Audio file not found: {audio_path.resolve()}")

backend = FoundryBackend()
with backend.model("whisper-tiny", print) as speech_model:
    transcript = backend.transcribe(speech_model, audio_path, "en")

if not transcript.strip():
    raise RuntimeError("No speech was recognized. Check the audio and language.")

print("\nTranscript:\n", transcript)
summary = backend.summarize(transcript, print)
print("\nDraft summary:\n", summary)

Run it while Flask and other inference jobs are stopped:

python transcribe_and_summarize.py

The context manager unloads Whisper before summarize() loads Qwen3. This script deliberately adds a file-to-summary flow; it does not change the file-upload tab's transcription-only behavior.

Summarization settings and longer recordings

The pinned sample uses a native ChatSession with an OpenAI-format JSON payload. These are the relevant generation settings, excerpted from that payload:

{
  "temperature": 0.2,
  "max_tokens": 700,
  "chat_template_kwargs": {
    "enable_thinking": false
  }
}

The prompt asks for English topics, key points, decisions, and action items, preserves names and numbers, and treats the transcript as data rather than instructions. The implementation disables Qwen's thinking mode explicitly, removes residual thinking markup, and reports an error for an empty answer. It does not substitute reasoning text for a summary.

For transcripts longer than 3,500 characters, the adapter summarizes sections and combines their notes recursively, for at most eight levels. If compression stops reducing the text, it preserves section summaries instead of silently truncating the source transcript. Character limits are an implementation heuristic, not a model-independent token budget.

Low temperature and a careful prompt do not guarantee faithful output or prevent all prompt-injection behavior. Keep the original transcript and require human review.

Explore the full implementation

Run the automated tests

From the sample repository root:

python -m pip install -r requirements-dev.txt
python -m pytest tests -q -rs

Most tests use fake inference backends. Native buffer tests require the standard SDK; browser regression tests require Chromium and can otherwise be skipped. Inspect the skip reasons. A passing test run does not establish real-model accuracy, microphone functionality, or catalog availability on your Pi.

Recorded Raspberry Pi benchmark results

The public benchmark results contain four complementary experiments: Whisper file transcription, English live transcription, Chinese live transcription, and Qwen3 summarization. The reports describe measurements made on October 19, 2026. The common platform was a Raspberry Pi 5 Model B Rev 1.1 with 8 GB RAM and four Cortex-A76 cores, running Linux ARM64 and Foundry Local SDK 2.1.0 with CPUExecutionProvider. The reports record Python 3.14.8; use the supported interpreter/wheel combination available for your own installation.

Each model or scenario ran three times in a fresh process, with models already downloaded and no concurrent inference. Unless stated otherwise, times are medians, parenthesized intervals are observed minimum-to-maximum ranges, and memory is the maximum process peak RSS across the runs. Operating-system file caches and desktop background workloads were not cleared. Fresh processes are not equivalent to cold-machine tests.

1. Whisper: English audio-file transcription

The Whisper report uses the original, unmodified 16.488-second, 24 kHz stereo MP3 with language auto-detection. The runtime handled decoding and preprocessing inside the measured transcription call.

The resolved variants were openai-whisper-tiny-generic-cpu:4 and openai-whisper-small-generic-cpu:4.

Real-time factor = transcription time / audio duration. The medians correspond to approximately 6.16 times and 1.59 times real-time batch processing speed. Tiny transcribed this clip about 3.89 times faster; Small used about 2.43 times as much peak process memory.

Small's text was more fluent in a text-only review, but there was no verified reference transcript or listening-based accuracy assessment. Do not infer word error rate, character error rate, or a general accuracy ranking from that observation. HTTP upload, artifact saving, summarization, and model unload are outside these timing boundaries.

2. English and Chinese live transcription

The English live report and Chinese live report exercise the real NativeLiveSession and the application's transcript-assembly logic. They do not include browser capture, Flask HTTP, UI polling, or summary generation.

• English input: The same 16.488-second MP3, decoded with FFmpeg to 16 kHz mono PCM16LE before timing. The harness sent 527,616 bytes in 165 chunks at approximately real-time pacing.
• Chinese input: A 15.998563-second, 16 kHz mono PCM16 WAV. The harness extracted its original PCM without resampling and sent 511,954 bytes in 160 chunks. Language was explicitly zh-CN, not auto-detection.
• Pacing: A full chunk was 100 ms; the final shorter chunk was preserved. No additional trailing silence was inserted. Stream timing began immediately before the first audio append.

The Chinese final transcripts were identical across the three runs and contained 74 Chinese characters and two <zh-CN> markers. The first nonempty confirmed event already contained Chinese characters, so the first-Chinese-character metric was not shortened by a language tag arriving alone.

The report flags wording in the raw output for human checking rather than silently correcting it. The English output also included awkward wording, such as You uh ban could give their fans. Successful streaming and repeatable output do not establish recognition accuracy. Preserve raw language tags and transcripts if you add display-layer cleanup.

3. Qwen3: English summarization of a completed transcript

The Qwen3 report uses a 786-character English transcript containing language tags. Its subject is a trip to New York, not the classroom sensor example above. Each run called the real FoundryBackend.summarize() path with qwen3-0.6b-generic-cpu:4, temperature=0.2, max_tokens=700, and enable_thinking=false.

The 14.052-second figure measures processing after the model is ready: prompt/payload construction, the native request, response parsing, and cleanup. It excludes model loading and unloading and is not a token-decoding-kernel measurement.

The 26.842-second value includes the model context inside summarize(), but the harness had already initialized the SDK and queried metadata. The 90.140-second subprocess measurement additionally includes harness setup, a separate catalog lookup, cleanup, telemetry, and result writing. It is not a measured Flask/browser interaction time or pure inference time. These timing boundaries differ, and medians of separate stages need not sum to a combined median.

Low temperature did not make the outputs identical. The report notes that some outputs were closer to sentence-by-sentence extraction and omitted details such as a travel companion or purchases. No human-reference summary score was calculated. Because the input was below 3,500 characters, this experiment does not validate long-transcript hierarchical summarization.

Troubleshooting and considerations

Common issues

Edge-device and responsible-use considerations

• CPU, not assumed acceleration: The recorded Pi used CPU execution. An ARM64 package does not automatically enable the Pi GPU or an add-on AI accelerator; compatible drivers, execution providers, and model variants are required.
• Power and thermals: Use active cooling and stable power. Where available, inspect vcgencmd measure_temp and vcgencmd get_throttled; other Linux installations may not provide these utilities.
• Storage and memory: A 60-minute, 16 kHz mono PCM16 recording contains approximately 115.2 MB of audio before its small WAV header. The streaming adapter also retains approximately that much PCM in memory until the session closes, in addition to model and object overhead.
• Persistence is not disaster recovery: Completed artifacts remain on disk, but data is not deleted automatically. Active sessions cannot resume after a process restart, and power loss can leave an incomplete recording.
• Local is not automatically secure: The sample does not provide user accounts, per-user authorization, or encryption at rest. Its request token protects against cross-site writes; it is not user authentication.
• Do not expose the demo as a classroom server: Binding to all interfaces is not a deployment strategy. A networked service needs HTTPS, authentication, authorization, retention policies, rate limits, and an appropriate application-serving architecture.
• Check network and privacy requirements: Inference is local, but model downloads, updates, and SDK diagnostics or telemetry are separate concerns. Review the Foundry Local privacy FAQ and applicable terms, and verify the actual network behavior required by your environment.
• Keep people in the loop: Use synthetic or consented recordings for teaching. Review names, numbers, decisions, and action items before sharing notes. This prototype is not a validated accommodation service or a tool for high-stakes assessment.

Next steps

Start with one microphone, a short English recording, and the three prepared models. Once that path is reliable, extend the project deliberately:

1. Compare model trade-offs. Use representative recordings and a human-verified reference to compare Whisper Tiny and Small on time, memory, and word or character error rate.
2. Explore multilingual reflections. Verify a multilingual model on the Pi, test each intended language, and distinguish transcription language from English summary language.
3. Improve note quality. Ask learners to compare summaries with transcripts, identify unsupported claims, and test whether a revised prompt actually improves factual fidelity.
4. Add retention controls. Implement an explicit delete/export workflow and consider operating-system storage encryption before using sensitive material.
5. Improve long-session robustness. Evaluate segmented recording, bounded native-session memory, and crash-safe artifact handling.
6. Design a different architecture for shared use. A multi-user classroom application needs more than extra Flask workers; reassess state management, authorization, and the inference runtime.

Useful resources:

• What is Foundry Local?
• Get started with Foundry Local
• Foundry Local architecture overview
• Tutorial: Build a voice-to-text note taker
• Live transcribe audio from a microphone
• Foundry Local CLI reference
• Complete Voice Notes community sample
• MDN: AudioWorklet
• MDN: Secure contexts