/enterprise/ Recent content on Tetragon - eBPF-based Security Observability and Runtime Enforcement Hugo en /docs/reference/daemon-configuration/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/reference/daemon-configuration/ Tetragon default controlling settings are set during compilation, so configuration is only needed when it is necessary to deviate from those defaults. This document lists those controlling settings and how they can be set as a CLI arguments or as configuration options from YAML files. Options The following table list all Tetragon daemon available options and is automatically generated using the tetragon binary --generate-docs flag. The same information can also be retrieved using --help. /docs/contribution-guide/development-setup/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/contribution-guide/development-setup/ Building and running Tetragon For local development, you will likely want to build and run bare-metal Tetragon. Requirements A Go toolchain with the version specified in the main go.mod; GNU make; A running Docker service (you can use Podman as well); The docker-buildx-plugin (you may already have this); For building tests, libcap and libelf (in Debian systems, e.g., install libelf-dev and libcap-dev). Build everything You can build most Tetragon targets as follows (this can take time as it builds all the targets needed for testing, see minimal build): /docs/concepts/events/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/events/ Tetragon’s events are exposed to the system through either the gRPC endpoint or JSON logs. Commands in this section assume the Getting Started guide was used, but are general other than the namespaces chosen and should work in most environments. JSON The first way is to observe the raw json output from the stdout container log: kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f The raw JSON events provide Kubernetes API, identity metadata, and OS level process visibility about the executed binary, its parent and the execution time. /docs/concepts/tracing-policy/example/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/tracing-policy/example/ To discover TracingPolicy, let’s understand via an example that will be explained, part by part, in this document: Warning This policy is for illustration purposes only and should not be used to restrict access to certain files. It can be easily bypassed by, for example, using hard links. apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "fd-install" spec: kprobes: - call: "fd_install" syscall: false args: - index: 0 type: "int" - index: 1 type: "file" selectors: - matchArgs: - index: 1 operator: "Equal" values: - "/tmp/tetragon" matchActions: - action: Sigkill The policy checks for file descriptors being created, and sends a SIGKILL signal to any process that creates a file descriptor to a file named /tmp/tetragon. /docs/installation/kubernetes/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/kubernetes/ The recommended way to deploy Tetragon on a Kubernetes cluster is to use the Helm chart with Helm 3. Tetragon uses the helm.cilium.io repository to release the helm chart. Install To install the latest release of the Tetragon helm chart, use the following command. Note You can find the chart and its documentation with all available values for configuration in install/kubernetes/tetragon in the Tetragon repository. You can use any of the values and override them with --set KEY1=VALUE1,KEY2=VALUE2. /docs/use-cases/host-system-changes/linux-kernel-modules/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/host-system-changes/linux-kernel-modules/ Monitoring kernel modules helps to identify processes that load kernel modules to add features, to the operating system, to alter host system functionality or even hide their behaviour. This can be used to answer the following questions: Which process or container is changing the kernel? Which process or container is loading or unloading kernel modules in the cuslter? Which process or container requested a feature that triggered the kernel to automatically load a module? /docs/use-cases/linux-process-credentials/syscalls-monitoring/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/linux-process-credentials/syscalls-monitoring/ Tetragon can hook at the system calls that directly manipulate the credentials. This allows us to determine which process is trying to change its credentials and the new credentials that could be applied by the kernel. This answers the questions: Which process or container is trying to change its UIDs/GIDs in my cluster? Which process or container is trying to change its capabilities in my cluster? Before going forward, verify that all pods are up and running, ensure you deploy our Demo Application to explore the Security Observability Events: /docs/concepts/enforcement/persistent-enforcement/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/enforcement/persistent-enforcement/ This page shows you how to configure persistent enforcement. Concept The idea of persistent enforcement is to allow the enforcement policy to continue running even when its tetragon process is gone. This is configured with the --keep-sensors-on-exit option. When the tetragon process exits, the policy stays active because it’s pinned in sysfs bpf tree under /sys/fs/bpf/tetragon directory. When a new tetragon process is started, it performs the following actions: checks if there’s existing /sys/fs/bpf/tetragon and moves it to /sys/fs/bpf/tetragon_old directory; sets up configured policy; removes /sys/fs/bpf/tetragon_old directory. /docs/use-cases/process-lifecycle/process-execution/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/process-lifecycle/process-execution/ This first use case is monitoring process execution, which can be observed with the Tetragon process_exec and process_exit JSON events. These events contain the full lifecycle of processes, from fork/exec to exit, including metadata such as: Binary name: Defines the name of an executable file Parent process: Helps to identify process execution anomalies (e.g., if a nodejs app forks a shell, this is suspicious) Command-line argument: Defines the program runtime behavior Current working directory: Helps to identify hidden malware execution from a temporary folder, which is a common pattern used in malwares Kubernetes metadata: Contains pods, labels, and Kubernetes namespaces, which are critical to identify service owners, particularly in a multitenant environments exec_id: A unique process identifier that correlates all recorded activity of a process As a first step, let’s start monitoring the events from the xwing pod: /docs/getting-started/install-k8s/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/getting-started/install-k8s/ Create a cluster If you don’t have a Kubernetes Cluster yet, you can use the instructions below to create a Kubernetes cluster locally or using a managed Kubernetes service: GKE AKS EKS Kind The following commands create a single node Kubernetes cluster using Google Kubernetes Engine. See Installing Google Cloud SDK for instructions on how to install gcloud and prepare your account. export NAME="$(whoami)-$RANDOM" export ZONE="us-west2-a" gcloud container clusters create "${NAME}" --zone ${ZONE} --num-nodes=1 gcloud container clusters get-credentials "${NAME}" --zone ${ZONE} The following commands create a single node Kubernetes cluster using Azure Kubernetes Service. /docs/troubleshooting/sysdump/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/troubleshooting/sysdump/ Before you report a problem, make sure to retrieve the necessary information from your cluster. Tetragon’s bugtool captures potentially useful information about your environment for debugging. The tool is meant to be used for debugging a single Tetragon agent node but can be run automatically in a cluster. Note that in the context of Kubernetes, the command needs to be run from inside the Tetragon Pod’s container. Key information collected by bugtool: /docs/use-cases/process-lifecycle/advanced-process-execution/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/process-lifecycle/advanced-process-execution/ Monitor ELF or Flat binaries execution Advanced process execution can be performed by using Tracing Policies to monitor the execve system call path. If we want to monitor execution of Executable and Linkable Format (ELF) or flat binaries before they are actually executed. Then the process-exec-elf-begin tracing policy is a good first choice. Note The process-exec-elf-begin tracing policy, will not report the different binary format handlers or scripts being executed, but will report the final ELF or flat binary, like the shebang handler. /docs/concepts/tracing-policy/argument_types/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/tracing-policy/argument_types/ Each argument definition specifies data type to be retrieved from kernel argument. The list contains simple POD types and several complex kernel objects that are represented by extracted data type. List of described data types: sint8, int8 uint8 sint16, int16 uint16 int, sint32, int32 uint32 long, sint64, int64 ulong, uint64, size_t string skb sock char_buf char_iovec filename fd cred const_buf nop bpf_attr perf_event bpf_map bpf_prog user_namespace capability kiocb iov_iter load_info module syscall64 kernel_cap_t cap_inheritable cap_permitted cap_effective linux_binprm data_loc net_device sockaddr socket file dentry path Note All integer types (int8, uint8, int16, uint16, int32, uint32, int64, uint64) support Equal, NotEqual, GT, LT, and Mask operators in matchArgs or matchData. /docs/installation/container/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/container/ Install Stable versions To run a stable version, please check Tetragon quay repository and select which version you want. For example if you want to run the latest version which is v1.6.0 currently. docker run --name tetragon --rm -d \ --pid=host --cgroupns=host --privileged \ -v /sys/kernel/btf/vmlinux:/var/lib/tetragon/btf \ quay.io/cilium/tetragon:v1.6.0 Unstable-development versions To run unstable development versions of Tetragon, use the latest tag from Tetragon-CI quay repository. This will run the image that was built from the latest commit available on the Tetragon main branch. /docs/use-cases/filename-access/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/filename-access/ This page shows how you can create a tracing policy to monitor filename access. For general information about tracing policies, see the tracing policy page. There are two aspects of the tracing policy: (i) what hooks you can use to monitor specific types of access, and (ii) how you can filter at the kernel level for only specific events. Hooks There are different ways applications can access and modify files, and for this tracing policy we focus in three different types. /docs/reference/helm-chart/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/reference/helm-chart/ The Tetragon Helm chart source is available under github.io/cilium/tetragon/install/kubernetes/tetragon and is distributed from the Cilium helm charts repository helm.cilium.io. To deploy Tetragon using this Helm chart you can run the following commands: helm repo add cilium https://helm.cilium.io helm repo update helm install tetragon cilium/tetragon -n kube-system To use the values available, with helm install or helm upgrade, use --set key=value. Values Key Type Default Description affinity object {} crds.installMethod string "operator" Method for installing CRDs. /docs/concepts/tracing-policy/hooks/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/tracing-policy/hooks/ Tetragon can hook into the kernel using kprobes and tracepoints, as well as in user-space programs using uprobes. Users can configure these hook points using the correspodning sections of the TracingPolicy specification (.spec). These hook points include arguments and return values that can be specified using the args and returnArg fields as detailed in the following sections. Warning Hooking a system call can introduce time-of-check to time-of-use (TOCTOU) races when the relevant argument is a pointer to user-space memory. /docs/troubleshooting/loglevel/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/troubleshooting/loglevel/ When debugging, it might be useful to change the log level. The default log level is controlled by the log-level option at startup: Enable debug level with --log-level=debug Enable trace level with --log-level=trace Change log level on Kubernetes Warning The Pods of the Tetragon DaemonSet will be restarted automatically after changing the debug Helm value. It is possible to change the log level of Tetragon’s DaemonSet Pods by setting tetragon.debug to true. /docs/contribution-guide/making-changes/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/contribution-guide/making-changes/ Make sure the main branch of your fork is up-to-date: git fetch upstream git checkout main git merge upstream/main For further reference read GitHub syncing a fork documentation. Create a PR branch with a descriptive name, branching from main: git switch -c pr/${GITHUB_USERNAME_OR_ORG}/changes-to-something main Make the changes you want. Test your changes. Follow Development setup and Running tests guides to build and test Tetragon. Make sure that all new code is covered by unit and/or end-to-end tests where feasible. /docs/use-cases/linux-process-credentials/monitor-changes-at-kernel/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/linux-process-credentials/monitor-changes-at-kernel/ Monitoring Process Credentials changes at the kernel layer is also possible. This allows to capture the new process_credentials that should be applied. This process-creds-installed tracing policy can be used to answer the following questions: Which process or container is trying to change its own UIDs/GIDs in the cluster? Which process or container is trying to change its own capabilities in the cluster? In which user namespace the credentials are being changed? /docs/getting-started/install-docker/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/getting-started/install-docker/ Note This guide has been tested on Ubuntu 22.04 and 22.10 with respectively kernel 5.15.0 and 5.19.0 on amd64 and arm64 but any recent distribution shipping with a relatively recent kernel should work. See the FAQ for further details on the recommended kernel versions. Note that you cannot run Tetragon using Docker Desktop on macOS because of a limitation of the Docker Desktop Linux virtual machine. Learn more about this issue and how to run Tetragon on a Mac computer in this section of the FAQ page. /docs/use-cases/security-profiles/record-linux-capabilities/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/security-profiles/record-linux-capabilities/ When the kernel needs to perform a privileged operation on behalf of a process, it checks the Capabilities of the process and issues a verdict to allow or deny the operation. Tetragon is able to record these checks performed by the kernel. This can be used to answer the following questions: What is the capabilities profile of pods or containters running in the cluster? What capabilities to add or remove when configuring a security context for a pod or container? /docs/troubleshooting/bpf-progs-stats/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/troubleshooting/bpf-progs-stats/ This page shows you how to monitor BPF programs statistics. Concept The BPF subsystem provides performance data for each loaded program and tetragon exports that in metrics or display that in terminal in top like tool. In terminal The tetra command allows to display loaded BPF programs in terminal with: tetra debug progs The default output shows tetragon programs only and looks like: 2024-10-31 11:12:45.94715546 +0000 UTC m=+8.038098448 Ovh(%) Id Cnt Time Name Pin 0. /docs/getting-started/execution/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/getting-started/execution/ At the core of Tetragon is the tracking of all executions in a Kubernetes cluster, virtual machines, and bare metal systems. This creates the foundation that allows Tetragon to attribute all system behavior back to a specific binary and its associated metadata (container, Pod, Node, and cluster). Observe Tetragon execution events Tetragon exposes the execution events over JSON logs and GRPC stream. The user can then observe all executions in the system. /docs/reference/grpc-api/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/reference/grpc-api/ The Tetragon API is an independant Go module that can be found in the Tetragon repository under api. The version 1 of this API is defined in github.com/cilium/tetragon/api/v1/tetragon. tetragon/bpf.proto BpfCmd Name Number Description BPF_MAP_CREATE 0 Create a map and return a file descriptor that refers to the map. BPF_MAP_LOOKUP_ELEM 1 Look up an element with a given key in the map referred to by the file descriptor map_fd. BPF_MAP_UPDATE_ELEM 2 Create or update an element (key/value pair) in a specified map. /docs/use-cases/network-observability/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/network-observability/ To view TCP connect events, apply the example TCP connect TracingPolicy: kubectl apply -f https://raw.githubusercontent.com/cilium/tetragon/main/examples/tracingpolicy/tcp-connect.yaml To start monitoring events in the xwing pod run the Tetragon CLI: kubectl logs -n kube-system -l app.kubernetes.io/name=tetragon -c export-stdout -f | tetra getevents -o compact --namespace default --pod xwing In another terminal, start generate a TCP connection. Here we use curl. kubectl exec -it xwing -- curl http://cilium.io The output in the first terminal will capture the new connect and write, /docs/concepts/tracing-policy/options/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/tracing-policy/options/ It’s possible to pass options through spec file as an array of name and value pairs: spec: options: - name: "option-1" value: "True" - name: "option-2" value: "10" Options array is passed and processed by each hook used in the spec file that supports options. At the moment it’s availabe for kprobe and uprobe hooks. Kprobe Options: options for kprobe hooks. Uprobe Options: options for uprobe hooks. Kprobe options disable-kprobe-multi: disable kprobe multi link disable-kprobe-multi This option disables kprobe multi link interface for all the kprobes defined in the spec file. /docs/installation/package/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/package/ Install Tetragon will be managed as a systemd service. Tarballs are built and distributed along the assets in the releases. Note Tetragon as of version 1.0 supports amd64 and arm64 architectures. First download the latest binary tarball, using curl for example to download the amd64 release: curl -LO https://github.com/cilium/tetragon/releases/download/v1.6.0/tetragon-v1.6.0-amd64.tar.gz Extract the downloaded archive, and start the install script to install Tetragon. Feel free to inspect the script before starting it. /docs/use-cases/process-lifecycle/privileged-execution/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/process-lifecycle/privileged-execution/ Tetragon also provides the ability to check process capabilities and kernel namespaces access. This information would help us determine which process or Kubernetes pod has started or gained access to privileges or host namespaces that it should not have. This would help us answer questions like: Which Kubernetes pods are running with CAP_SYS_ADMIN in my cluster? Which Kubernetes pods have host network or pid namespace access in my cluster? Step 1: Enabling Process Credential and Namespace Monitoring Edit the Tetragon configmap: /docs/contribution-guide/running-tests/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/contribution-guide/running-tests/ Tetragon has several types of tests: Go tests, composed of unit tests for userspace Go code and Go and BPF code. BPF unit tests, testing specifing BPF functions. E2E tests, for end-to-end tests, installing Tetragon in Kubernetes clusters and checking for specific features. Those tests are running in the Tetragon CI on various kernels1 and various architectures (amd64 and arm64). Go tests To run the Go tests locally, you can use: /docs/concepts/runtime-hooks/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/runtime-hooks/ Applying Kubernetes Identity Aware Policies requires information about Kubernetes (K8s) pods (e.g., namespaces and labels). Based on this information, the Tetragon agent can update the state so that Kubernetes Identify filtering can be applied in-kernel via BPF. One way that this information is available to the Tetragon agent is via the K8s API server. Relying on the API server, however, can lead to a delay before the container starts and the policy is applied. /docs/installation/runtime-hooks/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/runtime-hooks/ See Tetragon Runtime Hooks, for an introduction to the topic. Install Tetragon with Runtime Hooks We use minikube as the example platform because it supports both cri-o and containerd, but the same steps can be applied to other platforms. Setup cluster minikube with CRI-O minikube with Containerd kind (with Containerd) minikube start --driver=kvm2 --container-runtime=cri-o minikube start --driver=kvm2 --container-runtime=containerd Tetragon Runtime Hooks use NRI. NRI is enabled by default starting from containerd version 2. /docs/concepts/tracing-policy/selectors/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/tracing-policy/selectors/ Selectors enable per-hook in-kernel BPF filtering and actions. Each selector defines a set of filters as well as a set of (optional) actions to be performed if the selector filters match. Each hook can contain up to 5 selectors. If no selectors are defined on a hook, the default action (Post, i.e., post an event) will be used. Each selector comprises a set of filters: matchArgs: filter on the value of arguments. /docs/concepts/tracing-policy/tags/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/tracing-policy/tags/ Tags are optional fields of a Tracing Policy that are used to categorize generated events. Introduction Tags are specified in Tracing policies and will be part of the generated event. apiVersion: cilium.io/v1alpha1 kind: TracingPolicy metadata: name: "file-monitoring-filtered" spec: kprobes: - call: "security_file_permission" message: "Sensitive file system write operation" syscall: false args: - index: 0 type: "file" # (struct file *) used for getting the path - index: 1 type: "int" # 0x04 is MAY_READ, 0x02 is MAY_WRITE selectors: - matchArgs: - index: 0 operator: "Prefix" values: - "/etc" # Writes to sensitive directories - "/boot" - "/lib" - "/lib64" - "/bin" - "/usr/lib" - "/usr/local/lib" - "/usr/local/sbin" - "/usr/local/bin" - "/usr/bin" - "/usr/sbin" - "/var/log" # Writes to logs - "/dev/log" - "/root/. /docs/contribution-guide/documentation/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/contribution-guide/documentation/ Thank you for taking the time to improve Tetragon’s documentation. Find the content All the Tetragon documentation content can be found under github.com/cilium/tetragon/docs/content/en/docs. Note The main page served from a directory path is named _index.md. For example /docs/contribution-guide is available under /docs/content/en/docs/contribution-guide/_index.md. Style to follow We generally follow the Kubernetes docs style guide k8s.io/docs/contribute/style/style-guide. Preview locally To preview the documentation locally, use one of the method below. Then browse to localhost:1313/docs, the default port used by Hugo to listen. /docs/getting-started/file-events/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/getting-started/file-events/ Tracing policies can be added to Tetragon through YAML configuration files that extend Tetragon’s base execution tracing capabilities. These policies perform filtering in kernel to ensure only interesting events are published to userspace from the BPF programs running in kernel. This ensures overhead remains low even on busy systems. The instructions below extend the example from Execution Monitoring with a policy to monitor sensitive files in Linux. The policy used is file_monitoring. /docs/installation/tetra-cli/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/tetra-cli/ This guide presents various methods to install tetra in your environment. Install the latest release Autodetect your environment This shell script autodetects the OS and the architecture, downloads the archive of the binary and its SHA 256 digest, compares that the actual digest with the supposed one, installs the binary, and removes the download artifacts. Note This installation method requires a working Go toolchain, curl(1), and the sha256sum(1) utilities. For Go, see how to install the latest Go release and for the curl and checksum utility, it is usually distributed in common Linux distribution but you can usually find them respectively under the package curl and coreutils. /docs/concepts/tracing-policy/k8s-filtering/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/tracing-policy/k8s-filtering/ Motivation Tetragon is configured via TracingPolicies. Broadly speaking, TracingPolicies define what situations Tetragon should react to and how. The what can be, for example, specific system calls with specific argument values. The how defines what action the Tetragon agent should perform when the specified situation occurs. The most common action is generating an event, but there are others (e.g., returning an error without executing the function or killing the corresponding process). /docs/reference/metrics/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/reference/metrics/ Tetragon Health Metrics tetragon_bpf_error_metrics_total The total and type of errors encountered exposed via the BPF error metrics API. Internal use only. label values error 22 error_name EINVAL file_name bpf_d_path.h helper_func FnProbeRead line_number 166 tetragon_bpf_missed_events_total Number of Tetragon perf events that are failed to be sent from the kernel. label values error E2BIG, EAGAIN, EBUSY, EINVAL, ENOENT, ENOSPC, unknown msg_op 13, 14, 15, 16, 23, 24, 25, 26, 27, 28, 5, 7 tetragon_build_info Build information about tetragon /docs/use-cases/process-lifecycle/namespace-access/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/use-cases/process-lifecycle/namespace-access/ Tetragon can monitor Linux namespace operations to detect when processes change their namespaces. This is particularly useful for detecting container escapes where a process attempts to access host namespaces using tools like nsenter. Use Case: Detecting Container Escapes via Namespace Changes A common container escape technique involves using nsenter to switch into the host’s namespaces. Even if a container is running with restricted privileges, monitoring setns syscalls helps detect attempts to break out of the container’s isolation. /docs/concepts/tracing-policy/mode/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/tracing-policy/mode/ Beyond monitoring, Tetragon tracing policies include enforcement actions. Configuring the mode of a policy allows you to disable enforcement in a policy, without modifying the policy itself. A tracing policy can be set in two modes: monitoring: enforcement operations are elided enforcement: enforcement operations are respected and performed Using the tetra CLI, you can inspect the mode of each policy. For example: tetra tracingpolicy list Will produce out similar to: /docs/concepts/cgroup-rate/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/concepts/cgroup-rate/ This page shows you how to configure per-cgroup rate monitoring. Concept The idea is that tetragon monitors events rate per cgroup and throttle them (stops posting its events) if they cross configured threshold. The throttled cgroup is monitored and if its traffic gets stable under the limit again, it stops the cgroup throttling and tetragon resumes receiving the cgroup’s events. The throttle action generates following events: THROTTLE start event is sent when the group rate limit is crossed THROTTLE stop event is sent when the cgroup rate is again below the limit stable for 5 seconds Note The threshold for given cgroup is monitored per CPU. /docs/getting-started/network/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/getting-started/network/ In addition to file access monitoring, Tetragon’s tracing policies also support monitoring network access. In this section, you will see how to monitor network traffic to “external” destinations (destinations that are outside the Kubernetes cluster or external to the Docker host where Tetragon is running). These instructions assume you already have Tetragon running in either Kubernetes or Docker, and that you have deployed the Cilium demo application. Monitoring Kubernetes network access First, you’ll need to find the pod CIDR and service CIDR in use. /docs/contribution-guide/submitting-a-pull-request/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/contribution-guide/submitting-a-pull-request/ Note This guide is partially based on the Cilium contributing guide. Caution This guide assumes that you have already made and tested changes you want to contribute. If you have not, please follow the steps from the Contribution Guide. Commit changes Save your changes in one or more commits. If you are not comfortable with Git yet (in particular with git rebase), refer to the GitHub documentation. Caution Commits should separate logical chunks of code and not represent a chronological list of changes. /docs/reference/tracing-policy/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/reference/tracing-policy/ A TracingPolicy is a user-configurable Kubernetes custom resource (CR) that defines how Tetragon observes events in both the kernel and userspace using eBPF. It supports a variety of hook points including kprobes, uprobes, tracepoints, LSM hooks, and USDTs, giving users fine-grained control over what to trace and what actions to take. Policies consist of hook points, selectors for in-kernel filtering, and optional actions that can be executed when a match occurs. /docs/installation/verify/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/verify/ Verify Tetragon image signature Learn how to verify Tetragon container images signatures. Prerequisites You will need to install cosign. Verify Signed Container Images Since version 0.8.4, all Tetragon container images are signed using cosign. Let’s verify a Tetragon image’s signature using the cosign verify command: cosign verify --certificate-github-workflow-repository cilium/tetragon --certificate-oidc-issuer https://token.actions.githubusercontent.com <Image URL> | jq Note If you are using cosign < v2.0.0, you must set COSIGN_EXPERIMENTAL=1 environment variable to allow verification of images signed in KEYLESS mode. /docs/installation/configuration/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/configuration/ Depending on your deployment mode, Tetragon configuration can be changed by: Kubernetes Docker systemd kubectl edit cm -n kube-system tetragon-config # Change your configuration setting, save and exit # Restart Tetragon daemonset kubectl rollout restart -n kube-system ds/tetragon # Change configuration inside /etc/tetragon/ then restart container. # Example: # 1. As a privileged user, write to the file /etc/tetragon/tetragon.conf.d/export-file # the path where to export events, example "/var/log/tetragon/tetragon.log" # 2. Bind mount host /etc/tetragon into container /etc/tetragon # Tetragon events will be exported to /var/log/tetragon/tetragon. /docs/contribution-guide/developer-certificate-of-origin/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/contribution-guide/developer-certificate-of-origin/ To improve tracking of who did what, we’ve introduced a “sign-off” procedure, make sure to read and apply the Developer’s Certificate of Origin. /docs/getting-started/enforcement/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/getting-started/enforcement/ Tetragon’s tracing policies support monitoring kernel functions to report events, such as file access events or network connection events, as well as enforcing restrictions on those same kernel functions. Using in-kernel filtering in Tetragon provides a key performance improvement by limiting events from kernel to user space. In-kernel filtering also enables Tetragon to enforce policy restrictions at the kernel level. For example, by issuing a SIGKILL to a process when a policy violation is detected, the process will not continue to run. /docs/installation/metrics/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/metrics/ Tetragon exposes a number of Prometheus metrics that can be used for two main purposes: Monitoring the health of Tetragon itself Monitoring the activity of processes observed by Tetragon For the full list, refer to metrics reference. Enable/Disable Metrics Kubernetes In a Kubernetes installation, metrics are enabled by default and exposed via the endpoint /metrics. The tetragon service exposes the Tetragon Agent metrics on port 2112, and the tetragon-operator-metrics service the Tetragon Operator metrics on port 2113. /docs/contribution-guide/release-notes/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/contribution-guide/release-notes/ Tetragon release notes are published on the GitHub releases page. To ensure the release notes are accurate and helpful, contributors should write them alongside development. Then, at the time of release, the final notes are compiled and published. This guide is intended for both Tetragon developers and reviewers. Please follow it when creating or reviewing pull requests. release-note blurb in PR When you create a pull request, the template will include a release-note blurb in the description. /docs/contribution-guide/contributor-ladder/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/contribution-guide/contributor-ladder/ Tetragon is a sub-project of the Cilium project and follows the same governance model, community processes, and contributor growth paths as Cilium. To support contributors in gaining both privileges and responsibilities, we adopt the shared Contributor Ladder. This contributor ladder defines how contributors can grow from community participants to project maintainers, along with the expectations at each level. Community members generally start at the first levels of the ladder and advance as their involvement deepens. /docs/installation/faq/ Mon, 01 Jan 0001 00:00:00 +0000 /docs/installation/faq/ What is the minimum Linux kernel version to run Tetragon? Tetragon needs Linux kernel version 4.19 or greater. We currently run tests on stable long-term support kernels 4.19, 5.4, 5.10, 5.15 and bpf-next, see this test workflow for up to date information. Not all Tetragon features work with older kernel versions. BPF evolves rapidly and we recommend you use the most recent stable kernel possible to get the most out of Tetragon’s features. /features/execution-monitoring/ Mon, 01 Jan 0001 00:00:00 +0000 /features/execution-monitoring/ Understanding the intricate behavior of processes within their environments is an important requirement for many security teams. The sheer volume and complexity of modern applications and infrastructure can make detecting and responding to threats challenging. Traditional security tooling often struggles to provide detailed, real-time information on process creation, execution, and termination, making it difficult to detect anomalies. Identifying whether processes are spawning unexpectedly, running with escalated privileges, or exiting abnormally is critical but can be cumbersome without the right tools. /features/network-observability/ Mon, 01 Jan 0001 00:00:00 +0000 /features/network-observability/ In cloud native environments, network monitoring is essential but often disconnected from the processes and workloads driving the activity. Traditional observability typically stop at capturing connections, leaving questions about which processes initiated them or how they interact with other workloads and endpoints unanswered. This lack of context makes anomaly detection and root-cause analysis difficult. Tetragon brings clarity to network observability by directly linking every network event to the specific processes and workloads behind them. /features/kubernetes-identity-aware-policies/ Mon, 01 Jan 0001 00:00:00 +0000 /features/kubernetes-identity-aware-policies/ In traditional security environments, policies are often applied uniformly, assuming static infrastructure. This approach is ill-suited to Kubernetes clusters, where workloads are dynamic, ephemeral, and highly diverse. This complexity brings the need for security teams to enforce policies with precision. Enforcing policies too broadly may lead to excessive noise or performance overhead, while enforcing too narrowly can result in missing critical events. Tetragon addresses these challenges with Kubernetes Identity Aware Policies, leveraging eBPF for in-kernel filtering based on Kubernetes constructs like namespaces, pod labels, and container fields. /features/privileges-monitoring/ Mon, 01 Jan 0001 00:00:00 +0000 /features/privileges-monitoring/ In Linux environments, processes have various credentials—such as user and group IDs, capabilities, and security flags—that define their privileges. Threat actors frequently exploit these credentials to escalate privileges or circumvent security controls. Without real-time visibility into these changes, security teams are left blind to critical signs of compromise. Tetragon addresses this challenge by providing real-time monitoring of process credentials at both the system call and kernel levels. By hooking into system calls that manipulate credentials, Tetragon can capture detailed information about which processes or containers are attempting to change user IDs, group IDs, or capabilities. /features/file-integrity-monitoring/ Mon, 01 Jan 0001 00:00:00 +0000 /features/file-integrity-monitoring/ Security teams often face significant challenges in ensuring the integrity of critical files in their systems. Traditional file integrity monitoring (FIM) approaches, such as periodic file system scans, often miss rapid changes or fail to detect access events in real time. These methods are prone to delays, creating a window of opportunity for malicious actors to alter or access sensitive files without detection. Additionally, monitoring systems like Inotify can struggle with scalability, expressiveness, and the ability to filter events based on specific execution contexts, such as the identity of the process accessing the file. /features/operating-system-integrity/ Mon, 01 Jan 0001 00:00:00 +0000 /features/operating-system-integrity/ Modern containerized environments often rely on dynamic host system changes, such as loading kernel modules or modifying kernel parameters, to deliver necessary functionality. However, this same flexibility introduces significant security risks. Malicious actors can exploit these capabilities to load unauthorized kernel modules, alter system behavior, or hide their presence within the system. For security teams, the challenge lies in distinguishing legitimate activity from malicious manipulation of these critical kernel-level components. /features/capabilities-monitoring/ Mon, 01 Jan 0001 00:00:00 +0000 /features/capabilities-monitoring/ Processes running with high privileges or elevated capabilities can pose security risks, as they may bypass critical security controls or perform unauthorized actions. For example, misconfigured Kubernetes workloads might unintentionally request overly broad privileges, increasing the risk of exploitation. Identifying which processes are using elevated privileges and understanding the capabilities they require is essential for maintaining a secure environment. Security teams often struggle to identify and manage the precise capabilities required by applications running in Kubernetes environments and Linux capabilities are often granted too broadly due to a lack of visibility into what each workload truly needs. /search/ Mon, 01 Jan 0001 00:00:00 +0000 /search/