# KiwiSDR Integration
## Overview
The SDR Drone Pursuit System integrates with the KiwiSDR network to enhance signal detection and geolocation capabilities. KiwiSDR is a network of software-defined radios (SDRs) that provide remote access to RF spectrum data from various locations worldwide. This integration allows the system to:
- Fetch real-time signal data from multiple remote SDRs
- Perform geolocation using signal strength and SNR measurements
- Expand frequency coverage beyond local SDR hardware
## KiwiSDR Client
The `KiwiSDRClient` class in `kiwisdr_client.py` handles communication with the KiwiSDR network. It provides methods to:
1. Fetch and update the list of available KiwiSDR stations
2. Query specific stations for signal data at a given frequency
3. Aggregate measurements from multiple stations for enhanced analysis
### Key Features
- **Station List Management**: Automatically updates the list of active KiwiSDR stations every hour.
- **Frequency Matching**: Ensures that queried stations can receive the requested frequency.
- **Parallel Data Retrieval**: Uses asynchronous tasks to fetch data from multiple stations simultaneously.
- **Error Handling**: Logs errors and skips stations that fail to respond.
## Station List Management
The `update_station_list` method fetches the latest list of active KiwiSDR stations from the KiwiSDR API. It filters stations based on their online status and parses their frequency coverage.
```python
async def update_station_list(self, force: bool = False) -> None:
"""Update the list of available KiwiSDR stations"""
now = datetime.now().timestamp()
if not force and (now - self.last_update) < self.update_interval:
return
if not self.session:
raise RuntimeError("Client not initialized - use as context manager")
try:
async with self.session.get(self.station_list_url) as response:
if response.status == 200:
data = await response.json()
stations = []
for station in data.get('stations', []):
if station.get('status') == 'online':
stations.append(KiwiStation(
station_id=station['id'],
name=station['name'],
url=station['url'],
latitude=float(station.get('lat', 0)),
longitude=float(station.get('lon', 0)),
band_coverage=self._parse_band_coverage(station.get('bands', '')),
last_seen=now
))
# Update stations dict
self.stations = {s.station_id: s for s in stations}
self.last_update = now
logger.info(f"Updated KiwiSDR station list: {len(self.stations)} active stations")
else:
logger.error(f"Failed to fetch station list: HTTP {response.status}")
except Exception as e:
logger.error(f"Error updating station list: {e}")
```
## Querying Station Data
The `get_station_data` method retrieves signal data from a specific KiwiSDR station for a given frequency. It ensures that the frequency is within the station's coverage and returns details such as signal strength and SNR.
```python
async def get_station_data(self, station: KiwiStation, frequency: float) -> Optional[Dict]:
"""Get data from a specific KiwiSDR station for a given frequency"""
if not self.session:
raise RuntimeError("Client not initialized - use as context manager")
if not self._frequency_in_range(station, frequency):
return None
try:
url = f"{station.url}/api/data"
params = {
"freq": frequency/1e6, # Convert to MHz
"compression": "none",
"output": "json"
}
async with self.session.get(url, params=params) as response:
if response.status == 200:
data = await response.json()
return {
"station_id": station.station_id,
"station_name": station.name,
"latitude": station.latitude,
"longitude": station.longitude,
"frequency": frequency,
"signal_strength": data.get("signal_strength", 0),
"snr": data.get("snr", 0),
"timestamp": datetime.now().timestamp()
}
else:
logger.warning(f"Failed to get data from {station.name}: HTTP {response.status}")
return None
except Exception as e:
logger.error(f"Error getting data from {station.name}: {e}")
return None
```
## Aggregating Measurements
The `get_measurements` method collects signal data from multiple KiwiSDR stations for a given frequency. It prioritizes stations based on their last seen time and limits the number of queried stations to optimize performance.
```python
async def get_measurements(self, frequency: float, max_stations: int = 5) -> List[Dict]:
"""Get measurements from multiple KiwiSDR stations for a frequency"""
await self.update_station_list()
# Find stations that can receive this frequency
suitable_stations = [
station for station in self.stations.values()
if self._frequency_in_range(station, frequency)
]
# Sort by last seen time and limit number of stations
suitable_stations.sort(key=lambda s: s.last_seen or 0, reverse=True)
suitable_stations = suitable_stations[:max_stations]
# Get data from each station
tasks = [
self.get_station_data(station, frequency)
for station in suitable_stations
]
results = await asyncio.gather(*tasks)
return [r for r in results if r is not None]
```
## Frequency Coverage
The `_parse_band_coverage` and `_frequency_in_range` methods ensure that only stations capable of receiving the requested frequency are queried. This improves efficiency and accuracy.
```python
def _parse_band_coverage(self, bands_str: str) -> List[Dict[str, float]]:
"""Parse band coverage string into frequency ranges"""
coverage = []
try:
for band in bands_str.split(','):
if '-' in band:
start, end = band.split('-')
coverage.append({
'start': float(start),
'end': float(end)
})
except Exception:
pass
return coverage
def _frequency_in_range(self, station: KiwiStation, frequency: float) -> bool:
"""Check if a frequency is within a station's coverage"""
freq_mhz = frequency / 1e6
return any(
coverage['start'] <= freq_mhz <= coverage['end']
for coverage in station.band_coverage
)
```
## Integration with SDR System
The KiwiSDR client is integrated into the SDR Drone Pursuit System to:
1. Enhance geolocation accuracy by aggregating data from multiple remote stations.
2. Expand frequency coverage beyond the capabilities of local SDR hardware.
3. Provide redundancy in case of local hardware failure.
### Example Usage
```python
async def main():
async with KiwiSDRClient() as client:
frequency = 14.2e6 # 14.2 MHz
measurements = await client.get_measurements(frequency)
for measurement in measurements:
print(measurement)
if __name__ == "__main__":
asyncio.run(main())
```
## Web Interface Integration
The system now includes a user-friendly web interface for connecting to KiwiSDR stations. This interface allows users to:
1. Input KiwiSDR server connection details
2. Connect to specific KiwiSDR instances
3. Visualize signal data in the 3D environment
4. Monitor connection status
### KiwiSDR Modal
The system features a modal dialog for KiwiSDR connection settings:
- **Server Address**: Input field for the KiwiSDR server hostname or IP
- **Port**: Input field for the server port (default: 8073)
- **Frequency**: Input field for the target frequency in kHz
- **Status Area**: Displays connection status and error messages
- **Connect/Disconnect Buttons**: Control the connection state
The modal is accessible by clicking the "KiwiSDR" button in the control panel.
### Connection Process
When a user connects to a KiwiSDR instance:
1. The frontend sends a request to the `/api/connect-kiwisdr` endpoint with connection parameters
2. The backend creates a KiwiSDR client and establishes a connection
3. Data from the KiwiSDR is streamed through the WebSocket relay to the frontend
4. The 3D visualization displays the signal data in real-time
5. Connection status is reflected in both the modal and status indicators
### Backend Implementation
The FastAPI backend implements several endpoints for KiwiSDR management:
```python
@app.post("/api/connect-kiwisdr")
async def connect_kiwisdr(request: KiwiSDRConnectRequest, background_tasks: BackgroundTasks):
"""Connect to a KiwiSDR instance"""
global kiwisdr_client, kiwisdr_task
# Stop any existing KiwiSDR connection
if kiwisdr_task:
kiwisdr_task.cancel()
await asyncio.sleep(0.5) # Give it time to clean up
try:
# Create a queue for communicating with the websocket
websocket_queue = asyncio.Queue()
# Start the KiwiSDR streaming task
kiwisdr_task = asyncio.create_task(
kiwisdr_streaming_task(
request.server_address,
request.port,
request.frequency,
websocket_queue
)
)
return {
"success": True,
"message": f"Connected to KiwiSDR at {request.server_address}:{request.port} on frequency {request.frequency} kHz"
}
except Exception as e:
return {
"success": False,
"error": f"Failed to connect to KiwiSDR: {str(e)}"
}
@app.post("/api/disconnect-kiwisdr")
async def disconnect_kiwisdr():
"""Disconnect from KiwiSDR"""
global kiwisdr_client, kiwisdr_task
if kiwisdr_task:
kiwisdr_task.cancel()
kiwisdr_task = None
kiwisdr_client = None
return {"success": True, "message": "Disconnected from KiwiSDR"}
else:
return {"success": True, "message": "No active KiwiSDR connection to disconnect"}
```
The backend manages the KiwiSDR connection in a background task that:
1. Creates a KiwiSDR client instance
2. Connects to the specified server
3. Continuously fetches data at the specified frequency
4. Formats the data for visualization
5. Forwards the data through the WebSocket connection
### Frontend Implementation
The frontend JavaScript handles KiwiSDR connections with these functions:
```javascript
async function connectKiwiSDR(serverAddress, port, frequency) {
try {
const response = await fetch('/api/connect-kiwisdr', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
server_address: serverAddress,
port: port,
frequency: frequency
})
});
const data = await response.json();
if (data.success) {
serviceStatus.kiwiSdr = true;
updateStatusIndicators();
return { success: true, message: data.message };
} else {
return { success: false, error: data.error };
}
} catch (error) {
console.error("Error connecting to KiwiSDR:", error);
return { success: false, error: error.message };
}
}
async function disconnectKiwiSDR() {
// Implementation of KiwiSDR disconnection...
}
```
## Using the KiwiSDR Interface
To connect to a KiwiSDR station:
1. Click the "KiwiSDR" button in the SDR Sources section of the control panel
2. Enter the server address (e.g., `kiwisdr.example.com`)
3. Enter the port number (default: 8073)
4. Enter the frequency in kHz (e.g., 7100 for 40m amateur band)
5. Click "Connect"
6. Monitor the status in the modal and debug output area
7. To disconnect, click "Disconnect" in the modal or "Stop All" in the control panel
## Streaming Process
When connected, the KiwiSDR client:
1. Retrieves signal data from the specified station
2. Extracts frequency and signal strength information
3. Formats the data for visualization
4. Sends the data through the WebSocket relay at regular intervals
5. The 3D visualization updates to display the incoming signal data
## Error Handling
The KiwiSDR implementation includes robust error handling:
- Connection failures are reported in the modal status area
- Automatic cleanup of failed connections
- Graceful disconnection when closing the application
- Visual feedback of connection state through status indicators
## Future Enhancements
1. **Improved Error Handling**: Add retries for failed requests.
2. **Station Filtering**: Allow filtering by geographic region or signal quality.
3. **Data Caching**: Cache station data to reduce API calls.
4. **Advanced Geolocation**: Use TDOA or multilateration for precise signal source tracking.
5. **Multiple Stations**: Connect to multiple KiwiSDR stations simultaneously for comparative analysis.
6. **Frequency Scanning**: Add support for scanning frequency ranges.
7. **Preset Management**: Save and load favorite KiwiSDR stations and frequencies.
8. **Audio Integration**: Add WebAudio support to listen to signals while visualizing.
test_frontend.js, which, despite its .js extension, actually contains the HTML and JavaScript of the KiwiSDR web interface. This isn't a "test" file in the usual sense (like a unit test); it's the actual KiwiSDR client UI. Simplifying this is a substantial task, as it involves understanding the structure and purpose of a large, complex web application. Let's break down how to approach this, what can be simplified, and what cannot be easily simplified.
Understanding the File's Structure
This file is a complete, self-contained HTML document that, when loaded in a web browser, displays the KiwiSDR user interface. It's composed of:
HTML ( and ): Defines the structure of the page:
: Contains metadata (title, character set), links to CSS stylesheets (kiwisdr.min.css), and includes JavaScript files (config.js, kiwisdr.min.js, and others). It also has a
KiwiSDR IQ Data Capture
Connecting to KiwiSDR...
Analyze kiwisdr.js (the unminified version):
Look for WebSocket: Find where the WebSocket connection is established. Look for code that uses new WebSocket(...). This is the entry point for communication.
Identify Message Handlers: Find the functions that handle incoming messages from the WebSocket. These will likely be event listeners attached to the WebSocket object (e.g., websocket.onmessage = function(event) { ... }).
Find IQ Data Processing: Within the message handlers, look for code that deals with data of type ArrayBuffer or similar. This is likely where the raw IQ data is received. The code will likely convert this binary data into JavaScript arrays or TypedArrays. This is the most important part to understand.
Identify Control Messages: Look for the code that sends messages to the server (e.g., websocket.send(...)). These messages will control the frequency, mode, and other settings. You'll need to understand the format of these messages.
Modify the JavaScript (in kiwi_iq_capture.html):
Remove UI Elements: Remove all the HTML elements (divs, canvases, buttons, etc.) that are related to the visual display. Keep only a minimal structure.
Call connect(): You'll need to call the appropriate functions in kiwisdr.js to initiate the connection. This usually involves creating an instance of the main KiwiSDR object and calling a connect() method (or similar). The exact names will depend on the unminified code.
Set Frequency and Mode: Call the functions in kiwisdr.js to set the desired frequency, modulation mode (IQ), and sample rate.
Implement a Data Handler: Create a JavaScript function that will be called every time a chunk of IQ data is received. This function will:
Receive the data (likely as an ArrayBuffer).
Convert the data to a suitable format (e.g., a NumPy array in your Python code, if you're using a bridge).
Crucially: You'll need a way to get this data out of the JavaScript environment and into your Python environment. Here are the most common approaches:
WebSocket Server in Python: Your Python code could run a WebSocket server. The JavaScript code would connect to this server and send the processed IQ data. This is the most robust and flexible solution, but also the most complex to set up.
Write to a File (Less Ideal): The JavaScript could periodically write the accumulated IQ data to a file (using JavaScript's Blob and URL.createObjectURL APIs to create a downloadable file). Your Python script would then monitor this file and read the data. This is simpler to implement, but less efficient and potentially less reliable.
Message Passing (if using a framework like Electron): If you were embedding this HTML/JS in a desktop application framework like Electron, you could use inter-process communication (IPC) to send the data directly to your Python code.
Test and Iterate: This will be an iterative process. You'll need to experiment with the code, use your browser's developer tools (console, network tab) to inspect messages, and gradually build up the functionality you need.
Example: Adding a console.log to the _on_message handler (Illustrative)
Let's assume that after unminifying kiwisdr.min.js, you find a function that looks like this (this is just an example – the real code will be different):
JavaScript
// Inside kiwisdr.js (after unminifying)
function _on_message(event) {
if (event.data instanceof ArrayBuffer) {
// Process binary data (likely IQ data)
let data = new Uint8Array(event.data);
console.log("Received binary data:", data); // ADD THIS LINE
// ... (rest of the data processing code) ...
} else {
// Process text data (likely control messages)
let msg = JSON.parse(event.data);
console.log("Received JSON message:", msg);
// ... (rest of the message handling code) ...
}
}
By adding console.log, you can see the raw data being received in your browser's developer console. This will help you understand the format and how to process it.
Summary:
Simplifying the KiwiSDR frontend code for pure data acquisition is a process of:
Unminifying the core JavaScript (kiwisdr.min.js).
Removing the unnecessary HTML and JavaScript related to the visual interface.
Understanding the WebSocket communication and data formats.
Creating a minimal HTML file that loads the essential JavaScript and establishes the connection.
Implementing a data handler that receives and processes (or forwards) the IQ data.
Choosing a method to get the data from the JavaScript environment to your Python environment.
This process requires careful analysis of the existing code, but by focusing on the data acquisition parts and removing the UI elements, you can significantly reduce the complexity and create a streamlined data collection tool. The recommended approach is to use the RemoteSDRHandler and KiwiSDRClient to fetch the data in your python scripts.
Okay, you want to create a full-featured user interface, similar to the original KiwiSDR frontend. This is a significant undertaking, but a rewarding one if you want complete control and customization. Here's a breakdown of the approach, considerations, and steps involved:
Understanding the Goal
You're essentially aiming to rebuild a significant portion of the original test_frontend.js (which, remember, is actually the KiwiSDR client) while maintaining its core functionality. This means:
All UI Elements: You'll need to recreate the waterfall display, spectrum analyzer, control panels, frequency input, mode selection, audio controls, and all the other visual elements.
Full Functionality: The UI must be interactive and responsive, correctly controlling the KiwiSDR server and displaying the received data.
Maintainability: The code should be well-structured and easy to understand and modify.
Key Considerations
Complexity: This is a complex project. It requires a good understanding of HTML, CSS, JavaScript, WebSockets, and the KiwiSDR's communication protocol.
Time Commitment: Rebuilding the full UI will take a considerable amount of time and effort.
kiwisdr.js (Unminified): You'll be working very closely with the unminified version of kiwisdr.min.js. Understanding this code is crucial.
CSS Framework: The original KiwiSDR client uses W3.CSS. You can either continue using W3.CSS (recommended for consistency) or choose a different CSS framework (or write your own CSS). Using W3.CSS will save you a lot of time, as you can reuse the existing class names.
Modularity: Design your code with modularity in mind. Break down the UI into reusable components (e.g., a Waterfall component, a ControlPanel component, etc.). This will make the code easier to manage and extend.
Framework (Optional but Recommended): Consider using a JavaScript framework like React, Vue, or Angular. These frameworks help organize complex UIs, manage state, and improve performance. This adds a learning curve, but is highly recommended for a project of this scale. If you don't use a framework, you'll need to handle all the DOM manipulation and event handling manually, which can become very cumbersome.
Steps and Approach
Unminify: Make sure you have unminified versions of kiwisdr.min.js and kiwisdr.min.css.
Project Setup:
Create a new directory for your project.
Copy config/config.js into this directory.
Copy the unminified kiwisdr.js into this directory.
Create a new index.html file (this will be your main HTML file).
Create a style.css file (if you're not using W3.CSS directly, or if you need custom styles).
Create a script.js file (this will hold your custom JavaScript code, separate from kiwisdr.js).
Optional (but highly recommended): Set up a build process using a tool like Webpack, Parcel, or Rollup. This will help you manage dependencies, bundle your code, and potentially transpile your JavaScript (e.g., if you use a framework like React).
Basic HTML Structure (index.html):
HTML
My KiwiSDR Client
Added w3.css
Added script.js
Analyze and Recreate UI Elements (Iterative Process):
Start with the Major Containers: Begin by copying the main container divs from the original test_frontend.js into your index.html (inside the #app div). These are things like:
id-kiwi-container
id-top-container
id-main-container
id-waterfall-container
id-panels-container
id-control (the main control panel)
...and so on.
Add content to #app Copy the contents of the original test_frontend.js's body into the #app div of your index.html.
Connect Event Handlers: The original test_frontend.js uses inline event handlers (e.g., onclick="someFunction()"). You should avoid inline handlers in your new code. Instead, use JavaScript to attach event listeners to your elements. You can do this in your script.js file. For example:
JavaScript
// Inside script.js
function connectKiwi() {
//... your connection logic ...
}
document.addEventListener('DOMContentLoaded', function() {
// Get the button element
let connectButton = document.getElementById('id-connect-button'); // Assuming you gave your button this ID
// Add an event listener
if (connectButton) { // Check if the element exists
connectButton.addEventListener('click', connectKiwi);
}
// ... do the same for other buttons and controls ...
});
Move the connection logic inside your script.js file.
Canvas Elements: The waterfall and spectrum displays use