About the Micro-Agent Properties File

This document describes flags in the Java Micro-Agent properties and the use cases where they can be used.

Note that the former takipi.properties file that included all flags and switches has been divided into agent.properties and collector.properties to differentiate between the components. The properties are backward compatible.

The Micro-Agent properties are defined in the agent.properties file. All OverOps parameters are configurable via all three directives:

The Micro-Agent/service settings has a defined, uniform hierarchy. The priority of the methods is as follows:

1. JVM Option
2. Environment Option:
   a. Single Variables - TAKIPI_APPLICATION_NAME
   b. Environment String - TAKIPI_ARGS (comma separated)
3. Properties file - TAKIPI_HOME\agent.properties

Basic Micro-Agent Features

This section describes the basic OverOps Micro-Agent properties that can be configured.

Naming the Application

The application name should represent the logical components that make up your application.
Example: api, app, services, etc. Refer to Naming your Application, Deployments and Server for more information.

-Dtakipi.application.name (-Dtakipi.name)

Configure the application name of the JVM.

🚧

The application name should represent the logical components that make up your application.
Example: api, app, services, etc.

-Dtakipi.server.name (-Dtakipi.machine.name)

Configure the server and hostname on which the JVM runs when using a Remote Collector. This is in order to differentiate it from the machine name of the Collector, which is a different machine.

🚧

In a typical monolithic web application the actual server IP or VM names are very familiar and has meaning to you, but if you are running on a kubernetes cluster and 1000s of containers running in pods and are going up and down in an highly elastic manner the IPs mean nothing. In those cases your Server Grouping should be named after logical group instances.
Example: prod-app1, prod-api1, prod-api2, prod-service1, prod-sql1, etc.

-Dtakipi.deployment.name

Set the name of the application deployment version currently running on the JVM, also see Naming your Application, Server, Deployment

Directories

-Dtakipi.resources.dir (-Dtakipi.resources.folder)

This folder is used when different OS users are running different JVMs on the same host that may cause a conflict of permissions for the Micro-Agent under the Takipi resources directory. Permissions conflicts often cause instabilities in monitoring of applications in the UI. This parameter avoids the conflict by ensuring that each user/JVM/agent writes to its own folder.

-Dtakipi.resources.size.limit (-DresourceFolderLimit)

The size limit (in bytes) for the temporary resources folder used by the Micro-Agent.

-Dtakipi.properties.file (-Dtakipi.props.file)

Enable teams to deploy a dedicated agent.properties file per application and point to a Micro-Agent. Often configured with -Dtakipi.resources.dir.

Remote Collector

-Dtakipi.collector.host (-Dtakipi.daemon.host)

When using a Remote Collector, introduce the Collector to the Micro-Agent (this is an IP address).

🚧

To configure the Collector both host and port are required.

-Dtakipi.collector.port (-Dtakipi.daemon.port)

When using a Remote Collector, introduce the Collector to the Micro-Agent.

🚧

To configure the Collector both host and port are required.

-Dtakipi.collector.passphrase (-Dtakipi.passphrase)

When using a Remote Collector, configure it with a passphrase to identify authorized Micro-Agents.

-Dtakipi.collector.keepalive (-Dtakipi.keep.alive)

When using a Remote Collector, set the interval (in seconds) for the keep alive mechanism that sends a heartbeat to the Collector.

-Dtakipi.master.endpoints (-Dtakipi.daemon.endpoints)

For high-availability, set the master endpoint according to which to route traffic as round-robin.

🚧

Important

• Collector high-availability is not supported on macOS.

• In addition, the property takipi.master.endpoints should only be used in scenarios where OverOps is responsible for "load balancing" requests across collectors for two or more collectors. If you're using an external load balancer for collectors (i.e., Nginx), the takipi.collector.host and takipi.collector.port should be used to specify the load balancer.

Advanced Micro-Agent Features

Gathering Snapshots

-Dtakipi.cart.factor

Allows you to increase the amount of information collected for each snapshot. The amount of data collected during snapshot is determined by an internal algorithm called the cart.

The valid values vary between: 0.1-4 (including decimals to one decimal place). A cart factor of 2 records twice as much as a normal hit. A cart factor of 0.5 records half as much as a normal hit.
Note that larger snapshots may result in larger overhead. A larger cart factor is trading more overhead for more information. This does not affect the number of snapshots taken or levels deep into the heap.

-Dtakipi.stats.interval

Allows to set the timeframe in seconds how often the statistic data is pushed to the backend.
-1 = disabled

-Dtakipi.parallax

Allows to capture the snapshot data of the transaction which has introduced the slowness related to the Automatic Timers. See Automatic Timers and Slowdown Snapshots for details.

-Dtakipi.support.logger.frameworks

Decides which logging frameworks from the OverOps default list will be instrumented. If not set, all frameworks are used.

This parameter can take more than one framework using the : (colon) separator. If the frameworks are invalid, none will be used.

Supported logging frameworks:
log4j - Apache Log4j
log4j2 - Apache Log4j 2
logback - Logback Classic Logger
jul - Java Util Logging
slf4j - SLF4J Simple Logger
acl - Apache Common SimpleLogger
akka - Akka Logger
jboss - JBoss Logger
tinylog - Tinylog Logger

-Dtakipi.cerebro.min.log.level

Allows controlling the minimum log level recorded in the log view to reduce log overhead. Possible values include:

LOG_LEVEL_TRACE
LOG_LEVEL_DEBUG
LOG_LEVEL_INFO
LOG_LEVEL_WARN
LOG_LEVEL_ERROR

-Dtakipi.cerebro.message.max.length

Allows setting the maximum length of a recorded log line in the log view. Any log message recorded for the log view is trimmed to avoid exceeding this limitation.

-Dtakipi.max.depth

Configures how deep into the heap to let the Micro-Agent to capture variables. This may accelerate reaching the cart limit, resulting in displaying less data in other methods in a snapshot.

-Dtakipi.max.stack.depth

Sets the maximum number of frames collected in a snapshot. Moves from top (location of exception) to bottom (near thread run). The limitation includes 3rd-party (blacklisted) frames.

-Dtakipi.max.string.length

Sets the maximum length of a recorded string variable. This may reduce performance: lower performance for capture of longer strings.

-Dtakipi.max.array.length

Sets the maximum number of captured array elements. This may reduce performance: lower performance for capture of longer arrays.

-Dtakipi.max.string.encoding.size

Limits the size of the string that the Micro-Agent captures. This means that the Micro-Agent captures only the prefix of strings that are longer than this parameter.

-Dtakipi.max.array.encoding.size

Limits the size of the array that the Micro-Agent captures. This means that the Micro-Agent captures only the prefix of arrays that are longer than this parameter.

-Dtakipi.one.hit.req

The JVM only evacuates one snapshot per event for the entire lifetime of the JVM. This property can help reduce overhead when having one snapshot is enough.

-Dtakipi.concurrency.level

The number of concurrent snapshots to record. Limits the maximum number of snapshots the Micro-Agent can collect simultaneously. Range: 1-5.

-Dtakipi.shutdown.gracetime (-DshutdownGraceTime)

The period of time the Micro-Agent waits before shutting down to allow the last asynchronous messages to be sent to the Collector.

Neo Agent

-Dtakipi.neo

Runs the Agent in Neo mode: the next generation Neo Agent improves system stability and performance. If you're using Java 10 or 11, the Neo Agent is implemented by default on your system

🚧

Deploying the OverOps Neo Agent on Java 6, 7, 8 or 9

If you want to use the Neo Agent on Java 6, 7, 8, or 9, the defaults are disabled, and you'll need to work with the OverOps team to implement the new agent.

Neo Agent takipi.neo.heapless.inst.pre.vm=path/to/java-executable Flag

In version 4.38 added a mode to the Neo Agent to instrument all the JVM startup classes when loaded using an outer process. This addition reduces the statistics overhead, and improves value and performance. The flag can be specified as argument or configuration but must have a value on AIX pointing to the Java executable path on the machine.The flag is enabled by default in Linux, and isn't supported in Windows or OSX.

Optimization Features

Advanced Optimization

-Dtakipi.optimize.methods

Excludes a method during code throttling. In some cases, 3rd-party code is responsible for a significant part of the Micro-Agent’s overhead. Excluding it specifically from exception handling significantly reduces overhead.

📘

Note

This action does not ignore the method completely. It is efficient only where the OverOps Micro-Agent is creating a significant overhead.

-Dtakipi.optimize.packages

Excludes a package during code throttling. 3rd-party code potentially increases overhead on the Micro-Agent. Excluding it specifically from exception handling significantly reduces overhead.

📘

Note

This action does not ignore the package completely. It is efficient only where the OverOps Micro-Agent is creating a significant overhead.

-Dtakipi.boot.time

Configures a time frame during JVM startup in which the Micro-Agent does not collect any information (snapshots or statistics) or perform any instrumentation. This increases boot times slowed down by the OverOps Micro-Agent.

-Dtakipi.disable.exception.handling.time

Configures a time frame during JVM startup in which the Micro-Agent does not collect any information (snapshots or statistics) or perform any instrumentation. Callback is triggered, but OverOps will return immediately on every call. This increases boot times slowed down by the OverOps Micro-Agent.

Source Attach

-Dtakipi.repository.url

Repository URL

-Dtakipi.repository.type

The type of Git repository
Options are: GITLAB and GITHUB

-Dtakipi.repository.ref

The branch, tag, or SHA of your source code

-Dtakipi.repository.sources.root

The root directory of your source packages
For details what this property refers to please see Attaching Git Repository the note about Understanding Sources Root

-Dtakipi.repository.credentials

The token name created in the settings

Micro-Agent Feature Flags

The properties in this section disable certain functions in the Micro-Agent.

-Dtakipi.no.mpa.rex

Disables the ignoring of classes with parameter annotations. When classes have parameter annotations OverOps ignores them, which prevents capturing data from these classes (for example, disables timers). Most annotations are actually harmless and should not be a problem for the annotation, so this flag can override this safety switch and try to annotate the classes regardless.

-Dtakipi.no.recursive.context

In certain applications, some methods are repeatedly called in the same call stack. This causes significant CPU overhead. This flag disables instrumenting a method multiple times per call stack.

-Dtakipi.no.locals

Excludes the variable state from the snapshots, to reduce overhead for diagnostics purposes. This flag turns off the Micro-Agent’s JVMTI capability to extract local variable data during snapshot encoding causing significant overhead.

-Dtakipi.no.ex.callback

Turns off Micro-Agent registration to the JVMTI exception callback, that causes severe overhead in the JVM. When set, no exceptions are displayed. For diagnostics purposes.

-Dtakipi.no.ex

Turns off Micro-Agent JVMTI exception callback capability that causes severe overhead in the JVM. When set, callback is triggered but returned immediately. For diagnostics purposes.

-Dtakipi.no.xmen

Disables instrumentation and tracking of logged errors and logged warnings as events. This does not affect the log view.

-Dtakipi.no.cyder

Disables instrumentation and tracking of cross-machine tale stitching for rare synchronized HTTP requests.

-Dtakipi.no.robinhood

Disables instrumentation and extraction of deployment names from servlet contexts.

-Dtakipi.no.rocky

Disables instrumentation and tracking of HTTP errors.

-Dtakipi.no.apollo

Disables instrumentation and tracking of custom SDK events.

-Dtakipi.no.fin

Disables handling to differentiate between catch clauses and finally clauses.
Note: Disabling this flag may cause inaccurate ‘catch frame’ and ‘catch line’ designation.

-Dtakipi.no.rich.hits

Rich hits are snapshots that contain significantly more data than standard snapshots. Encoding rich hits increases overhead, that this flag turns off. Disabling rich hits has a similar effect to supplying a ‘cart factor’ argument (see above).

-Dtakipi.no.tlpk

Disables beautification of thread-local MDC recording. When recording thread-local data during snapshots, OverOps extracts logger MDC data surgically and makes it presentable for the user in the UI. Disabling takipi.no.tlpk keeps recording all thread-local data, but MDC information is buried within other TLS data.

-Dtakipi.no.cerebro

Disables log statement capture for the log view. It potentially provides significant performance benefits. Log capture can affect Garbage Collection when logs are dense. This does not affect recording of logged errors and logged warnings as events.

-Dtakipi.no.trex

Disables all instrumentation across the board. This, in essence, invalidates OverOps (error rates, timers, etc), but can be used to diagnose performance issues resulting from bytecode instrumentation.

-Dtakipi.no.req.inv

Disables instrumentation and collection of event statistics. Snapshots are still taken, but the Dashboard will not have any statistics for events, nor will statistics be sent to StatsD.

-Dtakipi.no.overmind

Disables instrumentation and collection of JVM view information.

-Dtakipi.no.tf

Disables the machine code throttling mechanism. Disabling this flag significantly decreases performance, but may be necessary for diagnostics.

-Dtakipi.no.tfa

Disables an auxiliary mechanisms for cleanup that involves some bytecode instrumentation. For diagnostics purposes.

-Dtakipi.no.etl

Disables instrumentation and injection of log links into throwable messages. This does not affect injection of log links into logging statements. Exception Tiny Links are disabled by default.

-Dtakipi.etl

Enables instrumentation and injection of log links into throwable messages. This does not affect the injection of log links into logging statements. Currently, ETL is disabled by default and this flag is required to turn it on.

For JVM arguments, you simply specify the flag - no value is required.
In the properties file, you'll need to specify either =1 or =true

-Dtakipi.no.chronos

Disables timers.

-Dtakipi.no.contexts

Disables instrumentation and collection of entry point information. When this flag is disabled, the Entry Points column on the Dashboard only displays the method which threw the event.

-Dtakipi.no.indy.rex

Disables any bytecode instrumentation to classes that include a special bytecode instruction called invoke dynamic.

-Dtakipi.no.xtl

Disables injection of log links into log statements.This does not affect the collection of logged warnings/errors as events, or the injection of log links into throwable messages.

-Dtakipi.no.memento

Disables displaying log links for all events, even when no specific snapshot was taken for that instance, and the log link points to the most recent snapshot available.

-Dtakipi.no.lf

Disables log capture optimizations: stops the throttler on the Micro-Agent callbacks to logged errors/warnings.

-Dtakipi.no.dynamic.inst

Disables all dynamic class instrumentation:

• Disables a complementary mechanism needed for optimizations to make sure we evacuate snapshots properly in JIT compiled mode. This issue could be seen in rare edge cases. (no Salinger)
• Disables our timer mechanisms (Chronos & Parallax)
• Logged error and warning small performance optimization
• Metrics such as the Error Rate will not show.

📘

Equivalent to:

-Dtakipi.no.req.inv
-Dtakipi.no.contexts
-Dtakipi.no.lf
-Dtakipi.no.tfa
-Dtakipi.no.chronos

-Dtakipi.optimized.mode

Disables a group of performance-related features to determine what is causing performance issues.

📘

Equivalent to

-Dtakipi.no.xmen
-Dtakipi.no.req.inv
-Dtakipi.no.contexts
-Dtakipi.no.fin
-Dtakipi.concurrency.level=1

-Dtakipi.loader.limited.inst Flag

If you're using several logging frameworks for different applications within the same web container, add the flag -Dtakipi.loader.limited.inst to CATALINA_OPTS in /bin/catalina.sh or by adding it to CATALINA_OPTS / JAVA_TOOL_OPTIONS.

🚧

Important

This flag should be used with caution and only when the default behavior doesn’t work - the flag is not intended as a best practice. Before using the flag, make sure an engineer looks at the Agent logs to validate that this is indeed the issue.

Micro-Agent Verbose Logging

-Dtakipi.log.to.console (-Dtakipi.debug.logconsole)

Makes the Micro-Agent write its internal log messages to the standard output instead of to the bugtale_agent.XXXX.log file.

This flag adds verbose debugging to all logs whenever turned on as long as the server is on; to stop, you'll, need to restart the service without the flag (it is recommended to use this flag for a limited time to figure out problems).

-Dtakipi.log.file (-Dtakipi.logfile)

When using Remote Collector, the default Micro-Agent log is written in the folder. This solves communication issues between the Micro-Agent and the Remote Collector.
The log file must be in an existing path.

-Dtakipi.silent

Prints debug log statements to the standard output. When false, upon the first event captured, the Micro-Agent prints sanity log statements and a log link to an event.

takipi.short.url.prefix

This option lets you set the base URL for tinylinks locally on the fly; this setting is used by default in On-Premises installations.

Dormant Triggers

dormant

Creating a dormant file in the Micro-Agent resources folder, causes it to shut down at runtime. The shutdown is visible in the Micro-Agent log. You'll need to restart the JVM to return to normal operations.

dormant.slow

Creating a dormant.slow file in the Micro-Agent resources folder, induces a gradual shutdown over the course of 20 minutes. Mechanisms are shut down separately, constantly updating the Micro-Agent log on feature shutdown and diagnostics. You'll need to restart the JVM to return to normal operations.

Turning off the New Relic Integration (aka Kill Switch)

Use the following command to turn off the New Relic integration with OverOps:
-Dtakipi.no.newrelic.integration

Running HotSpot JVMs for Java 10, 11 (Neo) in Lite Mode

In version 4.46 we added an option to run in HotSpot JVMs in lite mode, which is now set as the default. This minimizes the size of the instrumentation and requires a lot less CPU cycles to execute.

Users can opt-out of lite mode using the following flag:

Entrypoint customization

The following table describes the agent properties / environment variables that are used for defining prioritized / deprioritized entrypoints. Read more Entry Point Customization

Notes -

• Class / Packages names are case sensitive
• : is used as multi-parameters separator
• both . and / can be used as package/class delimiter

Usage examples -

1. takipi.prioritized.classes=MyPackage.MyClassA:MyPackage/MyClassB (usage example for multi parameters and different delimiters)
2. takipi.prioritized.packages=com.company:MyPackage (usage example for multi parameters)
3. takipi.deprioritized.classes=MyPackage/MyClassA