Optimize FT8 Skimmer Experience
Optimize FT8 Skimmer Experience
The work was triggered by a discussion on the [NextGenSDRs] group about the skimmer experience on Web-888. People noticed that Web-888 decoded fewer spots compared to SDRangel or WSJT-X.
This is not surprising as decoding FT8 requires a lot of CPU horsepower. However, I still wanted to try to see how much we could improve. I tried to port wsjt-x to web-888, however it is too slow. It needs 15 seconds one cpu to finish decoding. There is no way I can use it support 12/13 channels. The decoder in SDRangle is even slower.
So I stick with FT8_lib which is using on web-888. All the following changes are included in the version 20241220. The source code is available in the Web-888 repository.
performance of ft8_lib decoding
I use recall, precision, and F1-score to measure performance, using WSJT-X generated spots as the baseline.
The following command is used to generate the ground truth. I assume all extra spots beyond what WSJT-X decoded are fake signals or noise. However, this is not always accurate. I have noticed that some extra spots are valid, and they are actually WSJT-X's missing.
jt9 -d 2 -8 test.wavThe FT8_lib can only decode less than 70% spots of wsjt-x before the work.
The following table shows the improvement on the changes I made.
| Change | Extra | Missed | Recall | Precision | F1 Score |
|---|---|---|---|---|---|
| Improve regression tests, use WSJT-X result as baseline | 14 (0.9%) | 480 (31.7%) | 68.3% | 98.2% | 80.6% |
| Add OSD decoder | 15 (1.0%) | 466 (30.8%) | 69.2% | 97.9% | 81.1% |
| Increase time oversampling from 2 to 4 | 18 (1.2%) | 442 (29.2%) | 70.8% | 97.7% | 82.1% |
| Enable OSD deep decode | 19 (1.3%) | 436 (28.8%) | 71.2% | 97.5% | 82.3% |
| Subtract encoded messages | 20 (1.3%) | 415 (27.4%) | 72.6% | 97.4% | 83.2% |
| Two passes decode | 24 (1.6%) | 403 (26.6%) | 73.4% | 96.9% | 83.5% |
| Lower candidate threshold to 5 | 23 (1.5%) | 402 (26.6%) | 73.4% | 97.0% | 83.6% |
| Increase # of candidates to 256 | 27 (1.8%) | 391 (25.8%) | 74.2% | 96.5% | 83.9% |
| Modify subtraction algorithm | 24 (1.6%) | 390 (25.8%) | 74.2% | 96.9% | 84.1% |
| Use float when computing SNR | 28 (1.8%) | 386 (25.5%) | 74.5% | 96.4% | 84.0% |
| Subtract in each freq_sub | 34 (2.2%) | 371 (24.5%) | 75.5% | 95.6% | 84.4% |
| Three passes while reducing # of candidates per pass | 31 (2.0%) | 370 (24.4%) | 75.6% | 96.0% | 84.6% |
| Filter out fake signals via message formatting check | 27 (1.8%) | 370 (24.4%) | 75.6% | 96.5% | 84.8% |
Precision avoids false positives
Out of all the FT8 signals the decoder identified as valid decodes, how many were actually correct? (Measures how accurate decoding are.)
Recall avoids false negatives
Out of all the FT8 signals that were actually present, how many did the decoder successfully decode? (Measures how good decoder is at catching all signals.)
F1-Score balances above two metrics.
A balance between precision and recall. It’s a single number that tells you how well the decoder performs overall in identifying FT8 signals.
Detailed Explanation of Changes
Add OSD Decoder
In original lib, only one LDPC Decoder is implemented, BP Decoder (Belief Propagation). The decoder is fast but OSD is able to beat it by couple of db sensitivity.
The OSD (Ordered Statistics Decoder) is a powerful method for improving error correction during signal decoding. By integrating the OSD decoder, we added an additional layer of robustness to the decoding process, enabling the detection and correction of signals that previously might have been missed. If one signal is failed to decode by BP decoder with a small # of errors, we will use OSD to retry decoding.
Deep OSD decoding leverages more exhaustive computations within the OSD framework. It calculates the probability of each bit, guesses to flip the bit, and runs the CRC checksum and message format check. This gives a few more spots but not much.
Subtract Encoded Messages
This algorithm refines the decoding process by subtracting already-decoded signals from the signal spectrum. By doing so, it reduces interference caused by strong signals, making it easier to decode weaker ones. This step is especially effective in crowded bands where overlapping signals can obscure faint transmissions.
The tricky part is how to implement the subtraction algorithm while ft8_lib didn't even record the original signals. I implemented the algorithm in the frequency domain to subtract the signal's level only. Compared to the algorithm used in WSJT-X, this doesn't require any additional memory and uses very low CPU power. I tuned multi times how to apply the substraction.
The next hard part is determining how much we need to subtract. I used an approximate algorithm to assume the noise is the lowest signal in the 8 frequency bins of the signal. The signal subtracts the delta between the signal and the noise.
The subtraction gives quite a big improvement in performance.
Tune Some Parameters
After the new algorithm was implemented, I realized the original defaults were no longer optimal. So I tuned the parameters of decoding a bit, which gave another boost.
After all changes, Web-888 can decode 75% additional spots. Of course, no free lunch. The CPU is a bit higher than before.
Potential future work
- Use phase information. We are calculate phase information for signals. We are not using it.
- Improve canidate selection algorithm. Since we have limit CPU, if we can determine a signal is not a valid FT8 signal earlier, it saves lots of CPU.