Working with LLMs

View Source

LLMs are a new technology, and the patterns on how best to leverage them evolve every day. It is also quite debatable whether it is a good idea to use them at all. Nothing in Ash will ever be predicated on the usage of these tools, but we do want to provide at least some level of base guidance on what we think are the best practices for those that do. This is also to help those who are interested in trying these tools but don't yet know where to start.

This guide is about working with LLM dev assistants, not about building LLM-related features or integrating them into your application. For that, see Ash AI.

Getting Support with LLM generated code

Please note that LLMs often hallucinate despite our best intentions. If you need help with something, and you come to our support channels, you must make it clear when the code you are asking for help with was generated by an LLM. You must first understand the code you've written yourself, and provide a detailed explanation of the code and the issue you are facing when requesting help. The discord and forums are not a place for others to debug LLM hallucinations.

What to know

To take advantage of LLMs, you will want to explore the following. You will have to make up your own mind on which avenues to explore and leverage in the following areas. This is essentially a "big list of stuff you should research on your own".

  • Language models - OpenAI, Anthropic, Gemini, etc. Choice of language model will make the most difference in your experience.
  • Agentic editors - Zed, Windsurf, Cursor
  • Agentic assistants - Claude Code, Aider, Codex
  • Rules files - The name of these files are often specific to the editor, but they essentially boil down to "stuff to put in the system prompt" to guide the LLM's behavior.

Tools

We suggest setting up, where applicable, the following tools:

  • Tidewave: Tidewave gives your LLMs the ability to interact with your running application. This can significantly improve the quality of the code generated by LLMs, and allows them to observe and interact with the running application, like reading logs and working with processes.
  • Ash AI: Ash AI contains tools for building AI enabled applications, but it also comes with a dev MCP server where we will experiment with tools similar to what tidewave offers, but tailored to Ash and the way we work.

Rules

We are working on establishing a pattern whereby packages can provide a usage-rules.md which you can then combine into your own rules file. The idea here is to democratize the process of building rules, allowing you to adopt well vetted and quality rules files from the maintainers of projects. This has only been done for a few packages so far.

To leverage these rules files, you can simply copy them yourself if you'd prefer something more manual, or you can use a new mix task provided by the usage_rules package to combine them into your own rules file.

Combine all of your (direct) dependencies usage rules

mix usage_rules.sync .rules --all

Pick specific dependencies

mix usage_rules.sync .rules \
  ash ash_postgres ash_phoenix ash_graphql ash_json_api ash_ai

You can replace the .rules file with your own current rules file, and it will be appended to the contents. Repeated calls will only replace the package rules contents of the file, not the whole file contents.

Only dependencies of your current project will be added, and any dependencies that don't have rules are skipped.

Guide it on tools/design

It's also important to guide your AI on overall design decisions, and which tools to use. The majority of Elixir code LLMs are trained on will be using Phoenix contexts and direct Ecto calls.

Here are some recommendations for the top of your rule file for an Ash-centric app.

General Guidance

## Ash First

Always use Ash concepts, almost never Ecto concepts directly. Think hard about the "Ash way" to do things. If you don't know, look for information in the rules & docs of Ash & associated packages.

## Code Generation

Start with generators wherever possible. They provide a starting point for your code and can be modified if needed.

## Logs & Tests

When you're done executing code, try to compile the code, and check the logs or run any applicable tests to see what effect your changes have had.

Tidewave

## Tools

Use Tidewave MCP tools, as they let you interrogate the running application in various useful ways.

- Never attempt to start or stop a Phoenix application. Tidewave tools work by being connected to the running application, and starting or stopping it can cause issues.
- Use the `project_eval` tool to execute code in the running instance of the application. Eval `h Module.fun` to get documentation for a module or function.
- Always use `search_package_docs` to find relevant documentation before beginning work.

AshAI

## Tools

- Use `list_generators` to list available generators.
- Always prefer to use generators as a basis for code generation, and then modify afterwards.
- If you have to run generator tasks, pass `--yes`.