What's New in IntelliJ Elixir 11.11.0

IntelliJ Elixir, the Elixir plugin for JetBrains IDEs (IntelliJ IDEA, RubyMine, WebStorm, etc), version 11.11.0 has been released. Let’s see what’s new.

Reference Resolution Round-up

There was a large gap between IntelliJ Elixir 11.10.0 on 2020-02-10 and IntelliJ Elixir 11.11.0 on 2020-02-06-04, but that time was spent well with a total of 193 commits. 11.11.0 was so large because the primary goal of the release was to get as many references resolving as possible for a large client project.

How reference resolving broke

In the 2020.2+ series for JetBrains IDE releases, JetBrains changed the navigation API so that it favored declaration instead of references. This seemed like an innocuous change for Java and Kotlin where it is clear what is a declaration and what is a usage because of languages keywords. For Bash, or languages where everything can be a macro like in Clojure and Elixir, this was a massive breaking change.

This change broke most of the Go To Declarations actions because almost all bare words and calls with parentheses are PsiNamedElements, which under the new rules, makes it always a declaration. Under the old rules, a PsiNamedElement was only a declaration if its references referred back to itself.

So how did I fix this? The new rules still allow for a PsiNamedElement to not be a declaration, but it requires implementing TargetElementEvaluatorEx2#isAcceptableNamedParent. I implemented that method to stop module attributes and (L)EEx templates assigns names from counting as functions with an @ operator in front.

For Aliases, the unspoken rules were more complicated. While debugging Go To Declaration actions, I noticed they were sensitive to the range in element. It was this, coupled with the docs for PsiReference#getRangeInElement, that helped me realize the Go To Declaration and Completion have a hidden requirement: References for things that behave like namespaces have to work this way.

PsiElement representing a fully qualified name with multiple dedicated PsiReferences, each bound
to the range it resolves to (skipping the '.' separator).


PsiElement text: qualified.LongName
PsiReferences:   [Ref1---]X[Ref2--]
where {@code Ref1} would resolve to a "namespace" and {@code Ref2} to an "element".

Instead of references for only the outermost QualifiableAlias, there is a reference for each right-most Alias at a given position. Instead of there only being a reference to App.Context.Schema in App.Context.Schema, there is now a reference to App in the App prefix, a reference to App.Context in Context in App.Context, and a reference to App.Context.Schema in Schema in App.Context.Schema. Not only is this more useful–being able to jump to parent namespaces–it also fixed issues with Go To Definition in the 2020 line of IDEs. This approach of using getRangeInElement to target the range of the right-most Alias, while the element was still the parent that contained, but did not go beyond the Alias, was tried after having references only on Aliases and not QualifiedAliases did not fix completion issues.

Fixing Alias resolution also fixed finding the functions and macros defined in those modules. Go To Declaration was broken in both places for qualified or import calls because the Alias had to resolve first.

Scoping

Because the client project was so large and a monorepo, it gave me the first chance to test having multiple Jetbrains Modules in a Project that weren’t all in the same umbrella application.

Not being in the same umbrella means that dependencies should not be shared, so Elixir Module resolution is now limited to the same JetBrains Module, which corresponds to a specific Elixir/Erlang application and its dependencies ^{1 2} .

Performance improvements

Since version 10.0.0 the ModularName index has existed to make Go To Class work for modulars (defmodules, defimpls, defprotocols, etc). That index, which was optimized to only have modular names and not also function names, is now used for Go To Declaration and Completion for Aliases.

Most uses of the indexes now use the StubIndex#processElements instead of #getElements, so that a giant list of elements isn’t generated before being filtered. ³

The completion for modules names will no longer try to decompile the module just to gets its name, which is already known from the compiled .beam.

Ecto

Ecto queries are their own unique syntax, with some things that look like functions not even being macros and variables being defined with in and brought into scope with ^. For this reason, IntelliJ Elixir 11.11.0 contains special support for Ecto queries.

Variables

The Ecto docs call user in from(user in User) a “reference variable”. References variable usages are resolved to the left operand of in inside of from/2 and *join keywords to from/2; join/3-5 used for expressions; select/2-3; and where/2-3.

Like the optional/1 and required/1 in map types, the assoc/2 is pseudo-function and not even documented in Ecto.Query.API, so no reference is created for it.

Query.API

Ecto.Query.API contains all the helper functions you can use in queries, which includes operators like ==, but also functions like coalesce/2. The secret of Ecto.Query.API though is that the functions aren’t actually called by the Ecto.Query code! Ecto.Query.API only exists to show documentation!. So, if the code isn’t actually called, how can we resolve to it? Well, IntelliJ Elixir is doing static analysis, so I can make it resolve where I think makes sense. I’ve done this for all the Ecto.Query.API functions and operators. Quick Documentation and Go To Definition will go to those docs-only definitions. This works in where macros, from(select: ...), from(order_by: ..., select/2-3, select_merge, distinct/2-3, group_by/2-3, having/2-3, and order_by/2-3.

fragment/1 is even more full of lies than the other docs-only definitions. fragment, as used in actual code is not 1-arity. It has a variable number of arguments to match the number of ? in the fragment: fragment("lower(?)", p.title) is fragment/2, but fragment("coalesce(?, ?)", unquote(left), unquote(right)) is fragment/3. All these calls need to resolve to the fragment/1 definition, so fragment is treated like special forms to allow it to have an arity interval of 0.... That’s right, fragment can actually have infinite arity the same as with or for.

Schemas

Resolving field calls in an

schema "users" do
  field :name, :string
end

is more complicated than you might think. field isn’t a direct reference the the field/3 macro in Ecto.Schema.

How field actually works in schema for Ecto.Schema

1. use Ecto.Schema
2. Ecto.Schema.__using__
3. import Ecto.Schema, only: [schema: 2, embedded_schema: 1]

Note that only the outer DSLs, schema and embedded_schema are available even though field/2 is defined in Ecto.Schema.

So, when you call schema ... do the following code is called

1. defmacro schema(source, [do: block])
2. schema(__CALLER__, source, true, :id, block)
3. defp schema(caller, source, meta?, type, block)
4. There’s a big prelude = quote do quote block

At the end of prelude there is

try do
  import Ecto.Schema
  unquote(block)
after
  :ok
end

Hey! An import Ecto.Schema, but prelude is just floating as a variable. At the end of defp schema(caller, source, meta?, type, block) is

quote do
  unquote(prelude)
  unquote(postlude)
end

So to statically analyze an Ecto.Schema module.

1. Resolve schema/2 to defmacro schema by walking the use, __using__, quote, and import.
2. Inside the schema/2 (or macros in general if you want to get fancy 💅 and support more DSLs),
3. Go into the body of the macro. If there’s a call, resolve it
4. Go into the called function
5. Look for a quote block at the end (the same as my current __using__ support)
6. If there’s a Call inside an unquote see if you can resolve it to a variable in addition to a call definition (which is already supported for Phoenix).
7. If it’s a variable, check its value. If it’s a quote, use the quote block handling.
8. In the quote block handling add support for try
9. Walk the try and see the import, walk the import to find Ecto.Schema.field/2

🎉

Phoenix

Phoenix use of use MyWebApp, which has always been tricky to walk statically because it uses apply to figure out which function to call based on which.

@doc """
When used, dispatch to the appropriate controller/view/etc.
"""
defmacro __using__(which) when is_atom(which) do
  apply(__MODULE__, which, [])
end

Prior to 11.11.0, which was ignored and instead all functions in the first argument to apply were walked for a matching function, but in now, the second is used to walk only the function with the matching name. This allows resolving usages of assign/3 to correct Plug.Conn.assign/3 or Phoenix.LiveSocket.assign/3 based on whether it is in a controller or LiveView and LiveComponent.

LiveView

Functions in *.html.leex template files will resolve to functions defined in the corresponding LiveComponent or LiveView modules.

Assigns in the templates set with assign/2 will resolve to update/2 keywords, any other function, assign/3 or assign_new/3. Assigns can also only be defined because of how the LiveView was included in a parent view, such live_component or live_modal. live_modal is a little weird in that it’s not actually defined in Phoenix.LiveView, but something generated for all projects as a helper.

Aliases in templates will resolve even through use calls in the LiveComponent or LiveView module, which means Routes helpers work.

We Need to Go Deeper

There are some assigns that are actually generated by the framework. For those, they are resolved to the place in the deps that they are defined, such as @live_action, @myself, or @inner_content.

@socket instead of going to the library source resolves to the last socket variable or call in a view module, so that Find Usages doesn’t think all the @socket are the same. Likewise, @flash is resolved to put_flash/3 calls.

My goal is to not allow anything that appears in the code to be a mystery and for everyone to be able to understand how their code works.

Module Attributes

Module attributes no longer need to be defined directly in the using module and can instead be resolved through use calls.

Types

Types previously had highlighting and @spec for a function would resolve to the function, but there wasn’t dedicated resolvers for Type. This is fixed by now faking the built-in types in the decompiled erlang.beam. By defining the types in decompiled :erlang.beam (even though they aren’t actually defined there), there is a shared location for all reference to point to and then check for Find Usages. Type declarations also resolve to the type guards using keywords after when.

With the better type resolution though, some syntax in types that aren’t actually types caused unresolved references, so required/1 and optional/1 in type maps don’t generate references anymore.

The type t defined by defprotocol is resolved where defmacro defprotocol is defined if the SDK source is available, but the protocol-specific one if decompiled source is available for the specific protocol’s .beam.

Decompilation

EEP-48 includes documentation for types, so they are now decompiled too.

Functions and Macros

While defdelegate was supported in the Structure View previously, now definitions can be resolved through them too.

Resolving qualified and unqualified calls was unified, so use calls in the qualifier are walked and not just when in that actual module. To support functions that are defined at compile time, but in modules that have source modules can resolve to both the source and decompiled version when searching for functions or macros, but Go To Declaration will still favor source modules when clicking on their names.

Guards

defguard and defguardp are treated as defining calls.

Best-effort name/arity

If the qualifier for a qualified call can’t be resolved, a best effort match is made based on the call’s name and arity. It won’t always be right, but it may help you find the function or macro you’re looking for and the correct module name to use to get it. This is similar to how JetBrain’s own support for Java and Kotlin will suggest namespace corrections while treating the function name as correct.

Variables

The generated MyApp.Endpoint for mix phx.new has a section to enable code-reloading at compile time:

  # Code reloading can be explicitly enabled under the
  # :code_reloader configuration of your endpoint.
  if code_reloading? do
    socket("/phoenix/live_reload/socket", Phoenix.LiveReloader.Socket)
    plug(Phoenix.LiveReloader)
    plug(Phoenix.CodeReloader)
  end

Previously, code_reloading? variable would not resolve because psi.scope.Variable ignored use calls, now use calls are entered and the var!(code_reloading?) is found in Phoenix.Endpoint.config/1 by way of the unquote(config(opts)) call in the quote block in __using__(opts).

Reporting missing resolution

You can run the Elixir References Inspection by name to places where IntelliJ Elixir can’t yet resolve a reference at all or where it can only resolve to invalid results. (Invalid is used because of the API name ResolveResult#isvalidResult.) Invalid results doesn’t mean the resolution is known to be wrong, but means that the plugin doesn’t currently have a way to know for sure the resolution is right. It is using a heuristic that may be wrong.

If you run the inspection, can post a public repository of it, and explain what it should resolve to, I may be able to add more reference resolutions to the round-up.

Decompiling

Users have been busy reporting more weird functions names that trip up the decompiler. false, nil, and true can now all be decompiled as unquoted function names as happens in :thrift_json_parser. :hipe_arm_encode contains and that doesn’t take 2 arguments, while :hipe_sparc_encode contains or without 2 arguments and :digraph_utils contains in without 2 arguments.

Documentation

Documentation will now show more than just the doc strings, including any @deprecated, @impl, or @spec and the actual function heads, as the patterns guards can be helpful for understanding the types a function supports.

Better errors

Since these errors were a bit of a mystery with only the function name, the error reporting has been improved to include the module name too.

Installation

The plugin itself is free and open source as is IntelliJ Community Edition. You can download IntelliJ and install the plugin inside the IDE.

DockYard ❤️ FOSS

I’m able to work on IntelliJ Elixir because of DockYard Days, a program which provides team members with dedicated time to devote to individual skills and community growth projects, such as contributing to open source.

Supporting IntelliJ Elixir.

If you’d like, you can support the project directly.

If you have a complex Elixir project that you can’t open source like the client that helped drive this release, you can contact DockYard about having me work with your team to add missing features and the bug fixes you need to get the most out of your IDE.