Questions and answers

What it costs to run, what it records, where the recordings go, and how far to trust the answers, including what is not built or not measured yet.

Runtime Analysis is in early access. The recorder, the 27 tools on this site and the ra command line, MCP server and language server exist, and you can install them today. The hosted graph is built and not open to customers yet.

Running it in your process

What does it cost?

The cost has been measured on two test suites. Redmine’s has 5,917 tests that make 20 million recorded calls, and recording app and lib there added 16 percent to CPU time and 6 percent to the wall clock at tier 1, and 20 and 8 percent at tier 4. Campfire’s has 407 tests, and the cost there is about 1 percent, which is the amount two runs of the same configuration differ by and so too small to measure.

A server has been measured too, on our own machine and in nobody’s production. Redmine ran in production mode under Puma and answered 8 actions a second for 15 minutes. Recording at tier 4 added 9 percent to CPU time, and adaptive mode 3 percent. The slowest answer in a hundred took 185 ms at tier 4 and 178 ms in adaptive mode, against 170 ms unrecorded. No JIT was running. On a server whose JIT has compiled its hot paths, the recorder’s wrapper could be a larger share of each call, and that has not been measured.

The worst case is a loop calling one-line methods that do almost nothing, so the wrapper is nearly the whole cost. That loop runs 3.6 times slower when recording call edges and 5.5 times slower with argument classes and object ids, which is 98 to 168 nanoseconds added to a call. A method that builds a string, runs a query or renders a template does far more work per call, and the same fixed cost is a much smaller share of it.

Adaptive mode records a path in full the first time it appears and samples it after that. On Redmine’s suite at tier 4 it wrote 27 times less to disk and added 12 percent to CPU time where a full recording added 20. It saved no memory there. A server repeats its routes, so it gains more: under traffic the recording was 115 MB where the full one was 6.9 GB, and over the last five minutes the server held 35 MB more than an unrecorded one where a full recording held 104 MB more.

What recording costs has every figure and how it was measured, and it is the page we keep current.

What does it do to memory and garbage collection?

The native extension’s wrapper forwards your arguments without copying them. Each record is copied into a buffer the extension owns, outside Ruby’s heap, and a background thread encodes it and writes it to disk without holding Ruby’s global lock. A call the recorder has seen before is written by the C wrapper without calling into Ruby. A call it has not seen, a new call site or a method of an included gem goes through Ruby once and allocates. How the recorder works inside your process describes each step of this path.

On Redmine’s test suite, recording added 60 to 100 MB to a process that peaks at 457 MB unrecorded, at any tier. The memory grows with the number of distinct call routes the recorder has seen, at about 380 bytes each, and is independent of the number of calls, so it levels off. No setting bounds it. The collector ran for 6.5 seconds in an unrecorded run and 7.3 in one recorded at tier 1.

Can it take my app down?

The recorder is designed so that a fault in the tracer loses trace data and leaves your app running. Today it behaves that way in these cases:

When the buffer fills up, the record is dropped and counted, and the calling thread does not wait. In fifteen recorded runs of Redmine’s suite, one dropped a single row. When Campfire was recorded with every gem at tier 4, one run of three dropped 66,109 of 30.3 million rows. The six recorded runs of a server under traffic dropped none. A nearly full disk makes drops far more likely.
When an internal check fails, that one recording stream is switched off and a fresh one is opened.
When memory can’t be allocated, the record is dropped and counted.
When the disk write fails, the error goes to stderr and your app keeps running.
When your method raises, the exception reaches your code unchanged.

It does not yet behave that way in every case. In development we have seen a Rails server freeze with the gem enabled and not without it, under concurrent requests during a code reload, in an app using ActionCable’s PostgreSQL adapter. The tracer triggered the deadlock and was not itself part of it. We are removing every lock the recording thread can wait on and moving class wrapping out of the code-loading window. Until that is done and verified, record your test suite and development, and treat production as something to try with care.

Does it work with threads, forking servers and fibers?

Each process and thread gets its own recording stream, so threads don’t contend for one file, and after a fork the child opens its own streams. The call tree is tracked per fiber. Ractors are untested.

How does it hook into my code?

It prepends one module onto each class it records, and watches for new method definitions to do so. It does not use TracePoint to capture calls. We have not yet tested it alongside APM agents or other gems that prepend onto the same classes.

Methods defined before recording starts are not wrapped, so start it before your application code loads.

Do I have to add it to our Gemfile to try it?

No. After the one-line install, ra record runs your tests with the recorder loaded from the installed gem, and ra record dev does the same for your server. Your Gemfile, lockfile and helpers stay as they are, and git status shows nothing new: the recording goes to tmp/runtime_analysis, and the command you gave is saved under ~/.config/runtime_analysis. The free tools then answer from that recording, and after ra upgrade, ra activate installs the Pro build beside the free one without changing the project either. To record in CI you add the gem to the project, and the last step of ra setup offers those edits.

Can I turn it off, or sample?

RuntimeAnalysis.finish! stops recording in a running process. The wrappers stay in place and call your methods without recording. To sample, use adaptive mode. Starting some minutes after boot and recording on a share of servers are planned.

Which Rubies does it run on?

It runs on CRuby 3.2 and later. The test suite runs on 3.3, 3.4 and 4.0. Ruby 3.2 is past its end of life. The gem installs and records there, and we no longer test it. The C extension compiles at install and does all the recording. There is no pure-Ruby fallback, because a wrapper written in Ruby puts a frame of its own in front of each method, which changes what private and define_method do in the program being recorded. If the extension is not built, starting a recording says so and says how to build it.

It builds on Linux, with glibc or musl, and on macOS. Installing the gem needs a C compiler and make, which a slim container image leaves out: apt-get install build-essential on Debian, apk add build-base on Alpine. The extension is written against POSIX threads, so on Windows it needs WSL.

What gets recorded

Does it record every object and every message?

No. It records calls to methods defined in your own code, which by default means files under app and lib. You can add directories, and exclude directories or single files. Calls into gems, Rails and Ruby’s core classes are not recorded, so a call to String#upcase or Array#map writes no row. Configuring what gets recorded shows how to include a gem, and why gems are off by default.

What is written to disk?

You choose a tier, and each tier adds to the one before.

Tier 1 records, for each call, the caller’s and callee’s class name, method name, file path and line number, plus the file and line of the call site. Each call tree also gets an id, and a label you can set for where it started, such as a spec location or a job name.
Tier 2 adds the class name of each argument, and the names and classes of keyword arguments.
Tier 3 adds the object id of the receiver, each argument and the return value, and the return value’s class. An object id is an integer that identifies the same object wherever it appears within a run.
Tier 4 adds the class of an exception that passes through a call. It does not record the exception’s message or backtrace.

No tier records a value. Strings, numbers, attributes, SQL and the output of inspect are not written.

Aren’t class and method names sensitive too?

Yes. They describe your domain model, and file paths describe your layout. So recordings are written to your own disk, and nothing is sent anywhere unless you switch it on.

Where do the files go, and how big do they get?

They are written to ./tmp/runtime_analysis by default, as JSON lines, one file per process and thread. Every name is written once and referred to by number after that. You can rotate files by size, age or number of call trees, and cap the total kept. With no rotation configured a file grows for as long as the process runs, so set one for anything long-lived. Redmine’s whole suite came to 551 MB at tier 1, 1.4 GB at tier 4, and 53 MB at tier 4 in adaptive mode.

In a container the files are deleted with the container unless you mount a volume. Collecting them from short-lived containers is part of the hosted work and isn’t built.

Trusting the answers

My tests cover part of the app. What about the rest?

A recording contains what ran. A method that no recorded run reached has no callers, argument types or return shape in it. Recording your app as well as your test suite covers more of the code.

There is no report yet of which methods and which call sites were never observed. It is designed and not built.

How do I know a recording still matches the code?

Today you can’t tell without recording again. The planned design fingerprints each method’s signature and body, keeps the evidence for methods that haven’t changed, and marks the rest as unverified until they run again. It isn’t built yet.

Is an observed type complete?

An observed type lists the classes that arrived and the messages that were sent to them in the runs you recorded. A class that could arrive and never did in those runs is missing from it.

How is an object’s interface worked out?

From the messages sent to it, by inference over the call graph and your source. The inference is reliable within a method and less reliable once a value is passed between methods. A value that nothing is called on, or that passes through a method untouched, has no interface in the recording, so use it alongside static analysis.

How does it cope with metaprogramming?

Methods created with define_method are recorded like any other. A message handled by method_missing is recorded as a call to method_missing and is not listed under the name that was sent.

The hosted graph

The server that holds the hosted graph is built and running. It is not open to customers yet. What leaves your machine lists every field an upload contains.

What leaves my network?

On the free and Pro plans nothing leaves your network. On Team nothing leaves until you run the upload for a project. Before an upload is sent, every class and method name is replaced by a token, and the table that turns tokens back into names stays on your machines. The upload contains those tokens, the call counts between them, and the project name, commit sha and environment you recorded. It contains no file paths and no values.

Who can read it?

Today we can read what the server stores, which is a graph of tokens. The graph shows how many classes you have and which calls which, and it shows no names. The planned design encrypts the graph with a key held by your team, so that our servers store data they cannot read. That is designed and not built.

Is there a SOC 2 report, a DPA, an on-prem option?

There is no SOC 2 report and no DPA yet. Running the graph inside your own network is offered on the Enterprise tier and is arranged with you. There is no self-serve install.

How are different versions of my code kept apart?

Versions will be kept apart by the fingerprint of each method’s signature and body. Evidence is attached to one version of a method, so a new deploy keeps the evidence for the methods it did not change. This is designed and not built yet.

Asking

Are the tools available now?

Yes, on your own machine, and you can install them now. The gem has the recorder, every tool on the capabilities page plus observe, scaffold, refactor and check, the ra command line, ra mcp for a coding agent and ra lsp for an editor. Each tool page quotes its output on our example Rails app, and a script runs every quoted command and fails when a page differs from what the tool prints. What they found in Redmine shows the free tools on an app you can clone.

The hosted graph is built and is not open yet. These are not built: cypher over the hosted graph, editor quick-fixes, review comments on a pull request, and the diff against a base recording that ra check would need to hide findings already on main. The roadmap lists each of them.

Do the tools need my source code or production access?

They read the recording. Several also read your source on your machine, to connect a value returned in one place to the calls made on it in another. They do not connect to your running app.

Does my code go to an AI provider?

Runtime Analysis makes no calls to a language model. If you use it from a coding agent, the agent you chose sees the answers it asks for, which contain class and method names.

What does a tool say when the recording has no answer?

It reports that the recording has no answer. A stub whose real method did not run, for example, is reported as unverified.