Grafana Cloud

In the previous post, I shot, saved and viewed Opentelemetry data in Grafana Cloud.

If you use the free version of Grafana Cloud, you get about 50GB of logs and traces per month. If it is a service that does not accumulate traces (or does not record logs) because there are not many users, you can just use it, but if you introduce it on a small scale, I am afraid that too many logs will accumulate and explode.

Sampling

Sampling means extracting a part from the whole. As a result, the task is to reduce the number of stored telemetry data.

Why need sampling

Why is sampling necessary?

There is no need to save all the circles (trace) in the picture above. It is enough to store only important traces (errors, or too long execution time) and some samples representative of the whole (some of the OK traces).

Types of Sampling

Sampling can be broadly divided into Head Sampling and Tail Sampling.

Head Sampling

• Refers to sampling at the very beginning. A typical example is simply probabilistic sampling. Only 10% of the total trace is left and the rest is not traced.

Javascript

TraceIdRatioBasedSampler is provided by default.

import { TraceIdRatioBasedSampler } from '@opentelemetry/sdk-trace-node';

const samplePercentage = 0.1;

const sdk = new NodeSDK({
  // Other SDK configuration parameters go here
  sampler: new TraceIdRatioBasedSampler(samplePercentage),
});

disadvantage

• Sometimes important traces are dropped because they drop things without asking.

Tail Sampling

• Sampling from the back. At this time, since there is a lot of information available, you can filter it according to the desired logic.

• For example, error traces are always sampled.

• Usually, sampling is performed after all traces have been received from the collector.

disadvantage

• Implementation may be difficult. It is something that always needs to be changed when the system changes and conditions change.

• Difficult to perform because stateful state must be maintained for sampling.

• Tail Sampler may be vendor-specific.

avatar

Let’s implement Tail Sampling by implementing a Custom Span Processor.

SamplingSpanProcessor implementation

Create sampling-span-processor.ts file

import { Context } from "@opentelemetry/api";
import {
  SpanProcessor,
  ReadableSpan,
  Span,
} from "@opentelemetry/sdk-trace-node";

/**
 * Sampling span processor (including all error span and ratio of other spans)
 */
export class SamplingSpanProcessor implements SpanProcessor {
  constructor(
    private _spanProcessor: SpanProcessor,
    private _ratio: number
  ) {}

  /**
   * Forces to export all finished spans
   */
  forceFlush(): Promise {
    return this._spanProcessor.forceFlush();
  }

  onStart(span: Span, parentContext: Context): void {
    this._spanProcessor.onStart(span, parentContext);
  }

  shouldSample(traceId: string): boolean {
    let accumulation = 0;
    for (let idx = 0; idx  {
    return this._spanProcessor.shutdown();
  }
}

this._spanProcessor.onEnd(span); only when status.code is 2 (Error) or the ratio probability is winning. Export by calling

OtelSDK Update

Update spanProcessors in main.ts.

  spanProcessors: [
    new SamplingSpanProcessor(
      new BatchSpanProcessor(traceExporter),
      samplePercentage
    ),
  ],