abbyswag/qwen-code
1
0
Fork 0
mirror of https://github.com/QwenLM/qwen-code.git synced 2025-12-21 17:27:54 +00:00
Code Issues Packages Projects Releases Wiki Activity

Add GCP telemetry script (#1033)

Browse Source 
Adds a script -  `scripts/telemetry_gcp.js` - to simplify setting up a local OpenTelemetry collector that forwards data to Google Cloud. This is a follow up to the script for local telemetry `scripts/local_telemetry.js` added in #1015.

This script automates downloading necessary binaries, configuring the collector, and updating workspace settings.

Also includes `scripts/telemetry_utils.js` with shared helper functions for telemetry scripts. Will refactor `scripts/local_t elemetry.js` in next steps to use this shared functionality.

Updates `docs/core/telemetry.md` to include:
- A new "Quick Start" section
- Detailed instructions for the new GCP automated script
- Reorganization of existing sections for clarity

#750 

---
```
✨ Starting Local Telemetry Exporter for Google Cloud ✨
⚙️  Enabled telemetry in workspace settings.
🔧 Set telemetry OTLP endpoint to http://localhost:4317.
✅ Workspace settings updated.
✅ Using Google Cloud Project ID: foo-bar

🔑 Please ensure you are authenticated with Google Cloud:
  - Run `gcloud auth application-default login` OR ensure `GOOGLE_APPLICATION_CREDENTIALS` environment variable points to a valid service account key.
  - The account needs "Cloud Trace Agent", "Monitoring Metric Writer", and "Logs Writer" roles.
✅ otelcol-contrib already exists at /Users/jerop/github/gemini-cli/.gemini/otel/bin/otelcol-contrib
🧹 Cleaning up old processes and logs...
✅ Deleted old GCP collector log.
📄 Wrote OTEL collector config to /Users/jerop/github/gemini-cli/.gemini/otel/collector-gcp.yaml
🚀 Starting OTEL collector for GCP... Logs: /Users/jerop/github/gemini-cli/.gemini/otel/collector-gcp.log
⏳ Waiting for OTEL collector to start (PID: 65145)...
✅ OTEL collector started successfully on port 4317.

✨ Local OTEL collector for GCP is running.

📄 Collector logs are being written to: /Users/jerop/github/gemini-cli/.gemini/otel/collector-gcp.log

📊 View your telemetry data in Google Cloud Console:
   - Traces: https://console.cloud.google.com/traces/list?project=foo-bar
   - Metrics: https://console.cloud.google.com/monitoring/metrics-explorer?project=foo-bar
   - Logs: https://console.cloud.google.com/logs/query;query=logName%3D%22projects%2Ffoo-bar%2Flogs%2Fgemini_cli%22?project=foo-bar

Press Ctrl+C to exit.
^C
👋 Shutting down...
⚙️ Disabled telemetry in workspace settings.
🔧 Cleared telemetry OTLP endpoint.
✅ Workspace settings updated.
🛑 Stopping otelcol-contrib (PID: 65145)...
✅ otelcol-contrib stopped.
```
This commit is contained in:
Jerop Kipruto
2025-06-13 20:28:18 -07:00
committed by GitHub
parent defb0fac2c
commit 1452bb4ca4
3 changed files with 595 additions and 188 deletions
Download Patch File Download Diff File Expand all files Collapse all files

243
docs/core/telemetry.md
View File

@@ -6,6 +6,32 @@ This entire system is built on the **[OpenTelemetry] (OTEL)** standard, allowing
[OpenTelemetry]: https://opentelemetry.io/
## Quick Start
### Telemetry with Google Cloud
1. **Ensure Prerequisites:**
Ensure that:
- You have set the `GOOGLE_CLOUD_PROJECT` environment variable.
- You have authenticated with Google Cloud and have the necessary IAM roles.
For full details, see the [Google Cloud](#google-cloud) prerequisites.
2. **Run the Script:** Execute the following command from the project root:
```bash
./scripts/telemetry_gcp.js
```
3. **View Data:** The script will provide links to view your telemetry data (traces, metrics, logs) in the Google Cloud Console.
4. **Details:** Refer to documentation for telemetry in [Google Cloud](#google-cloud).
### Local Telemetry with Jaeger UI (for Traces)
1. **Run the Script:** Execute the following command from the project root:
```bash
./scripts/local_telemetry.js
```
2. **View Logs/Metrics:** Check the `.gemini/otel/collector.log` file for raw logs and metrics.
3. **View Traces:** Open your browser and go to `http://localhost:16686` to see traces in the Jaeger UI.
4. **Details:** Refer to documentation for telemetry in [Local](#local).
## Enabling Telemetry
You can enable telemetry in multiple ways. [Configuration](configuration.md) is primarily managed via the `.gemini/settings.json` file and environment variables, but CLI flags can override these settings for a specific session.
@@ -50,10 +76,9 @@ Learn more about OTEL exporter standard configuration in [documentation][otel-co
mkdir .gemini/otel
```
### Local (Automated Script)
### Local
For the most straightforward local setup, use the `scripts/local_telemetry.js` script. This script automates the entire process of setting up a local telemetry pipeline, including configuring the necessary settings in your `.gemini/settings.json` file.
The script installs `otelcol-contrib` (The OpenTelemetry Collector) and `jaeger` (The Jaeger UI for viewing traces). To use it:
Use the `scripts/local_telemetry.js` script that automates the entire process of setting up a local telemetry pipeline, including configuring the necessary settings in your `.gemini/settings.json` file. The script installs `otelcol-contrib` (The OpenTelemetry Collector) and `jaeger` (The Jaeger UI for viewing traces). To use it:
1. **Run the Script**:
Execute the script from the root of the repository:
@@ -81,202 +106,44 @@ The script installs `otelcol-contrib` (The OpenTelemetry Collector) and `jaeger`
4. **Stop the Services**:
Press `Ctrl+C` in the terminal where the script is running to stop the OTEL Collector and Jaeger services.
### Local (Manual Setup)
**1. Create a Configuration File**
Create the file `.gemini/otel/collector-local.yaml` with the following:
```bash
cat <<EOF > .gemini/otel/collector-local.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
processors:
batch:
timeout: 1s
exporters:
debug:
verbosity: detailed
service:
telemetry:
logs:
level: "debug"
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [debug]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [debug]
logs:
receivers: [otlp]
processors: [batch]
exporters: [debug]
EOF
```
**2. Run the Collector**
You can run the collector using `docker` or using the `otelcol-contrib` binary directly.
**_Option 1: Use Docker_**
This is the simplest method if you have Docker installed.
1. **Run the Collector**:
```bash
docker run --rm --name otel-collector-local \
-p 4317:4317 \
-v "$(pwd)/.gemini/otel/collector-local.yaml":/etc/otelcol-contrib/config.yaml \
otel/opentelemetry-collector-contrib:latest
```
2. **Stop the Collector**:
```bash
docker stop otel-collector-local
```
**_Option 2: Use `otelcol-contrib`_**
Use this method if you prefer not to use Docker.
1. **Run the Collector**:
Once installed, run the collector with the configuration file you created earlier:
```bash
./otelcol-contrib --config="$(pwd)/.gemini/otel/collector-local.yaml"
```
2. **Stop the Collector**:
Press `Ctrl+C` in the terminal where the collector is running.
### Google Cloud
This setup sends all telemetry to Google Cloud for robust, long-term analysis.
For a streamlined setup targeting Google Cloud, use the `scripts/telemetry_gcp.js` script which automates setting up a local OpenTelemetry collector that forwards data to your Google Cloud project.
**1. Prerequisites**
1. **Prerequisites**:
- A Google Cloud Project ID.
- **APIs Enabled**: Cloud Trace, Cloud Monitoring, Cloud Logging.
- **Authentication**: A Service Account with the roles `Cloud Trace Agent`, `Monitoring Metric Writer`, and `Logs Writer`. Ensure your environment is authenticated (e.g., via `gcloud auth application-default login` or a service account key file).
- Ensure you have a Google Cloud Project ID.
- Set the `GOOGLE_CLOUD_PROJECT` environment variable to your project ID.
- Authenticate with Google Cloud (e.g., run `gcloud auth application-default login` or ensure `GOOGLE_APPLICATION_CREDENTIALS` is set).
- Ensure your account/service account has the necessary roles: "Cloud Trace Agent", "Monitoring Metric Writer", and "Logs Writer".
**2. Set environment variables**
Set the `GOOGLE_CLOUD_PROJECT`, `GOOGLE_CLOUD_LOCATION`, and `GOOGLE_GENAI_USE_VERTEXAI` environment variables:
```bash
GOOGLE_CLOUD_PROJECT="YOUR_PROJECT_ID"
GOOGLE_CLOUD_LOCATION="YOUR_PROJECT_LOCATION" # e.g., us-central1
GOOGLE_GENAI_USE_VERTEXAI=true
```
**3. Create a Configuration File**
Create `.gemini/otel/collector-gcp.yaml`:
```bash
cat <<EOF > .gemini/otel/collector-gcp.yaml
receivers:
otlp:
protocols:
grpc:
endpoint: "0.0.0.0:4317"
processors:
batch:
timeout: 1s
exporters:
googlecloud:
project: "${GOOGLE_CLOUD_PROJECT}"
metric:
prefix: "custom.googleapis.com/gemini_cli"
log:
default_log_name: "gemini_cli"
debug:
verbosity: detailed
service:
pipelines:
traces:
receivers: [otlp]
exporters: [googlecloud]
metrics:
receivers: [otlp]
exporters: [googlecloud]
logs:
receivers: [otlp]
exporters: [googlecloud]
EOF
```
**4. Run the Collector**
You can run the collector for Google Cloud using either Docker or a locally installed `otelcol` binary.
**_Option 1: Use Docker _**
This method encapsulates the collector and its dependencies within a container.
1. **Run the Collector**:
Choose the command that matches your authentication method.
- **If using Application Default Credentials (`gcloud auth application-default login`)**:
```bash
docker run --rm --name otel-collector-gcp \
-p 4317:4317 \
--user "$(id -u):$(id -g)" \
-v "$HOME/.config/gcloud/application_default_credentials.json":/etc/gcp/credentials.json:ro \
-e "GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/credentials.json" \
-v "$(pwd)/.gemini/otel/collector-gcp.yaml":/etc/otelcol-contrib/config.yaml \
otel/opentelemetry-collector-contrib:latest --config /etc/otelcol-contrib/config.yaml
```
- **If using a Service Account Key File**:
```bash
docker run --rm --name otel-collector-gcp \
-p 4317:4317 \
-v "/path/to/your/sa-key.json":/etc/gcp/sa-key.json:ro \
-e "GOOGLE_APPLICATION_CREDENTIALS=/etc/gcp/sa-key.json" \
-v "$(pwd)/.gemini/otel/collector-gcp.yaml":/etc/otelcol-contrib/config.yaml \
otel/opentelemetry-collector-contrib:latest --config /etc/otelcol-contrib/config.yaml
```
2. **Check Status**:
Your telemetry data will now appear in Google Cloud Trace, Monitoring, and Logging.
3. **Stop the Collector**:
```bash
docker stop otel-collector-gcp
```
**_Option 2: Use `otelcol-contrib`_**
Use this method if you prefer not to use Docker.
1. **Run the Collector**:
2. **Run the Script**:
Execute the script from the root of the repository:
```bash
./otelcol-contrib --config="file:$(pwd)/.gemini/otel/collector-gcp.yaml"
./scripts/telemetry_gcp.js
```
2. **Check Status**:
Your telemetry data will now appear in Google Cloud Trace, Monitoring, and Logging.
The script will:
3. **Stop the Collector**:
Press `Ctrl+C` in the terminal where the collector is running.
- Download the `otelcol-contrib` binary if needed.
- Start an OTEL collector configured to receive data from the Gemini CLI and export it to your specified Google Cloud project.
- Automatically enable telemetry and disable sandbox mode in your workspace settings (`.gemini/settings.json`).
- Provide direct links to view traces, metrics, and logs in your Google Cloud Console.
- On exit (Ctrl+C), it will attempt to restore your original telemetry and sandbox settings.
---
3. **View Telemetry in Google Cloud**:
Use the links provided by the script to navigate to the Google Cloud Console and view your traces, metrics, and logs.
4. **Inspect Local Collector Logs**:
The script redirects the local OTEL collector's output to `.gemini/otel/collector-gcp.log`. You can monitor this file for detailed information or troubleshooting:
```bash
tail -f .gemini/otel/collector-gcp.log
```
5. **Stop the Service**:
Press `Ctrl+C` in the terminal where the script is running to stop the OTEL Collector.
## Data Reference: Logs & Metrics