ALLM

In building LLM assisted projects, I've found that Elixir is an ideal language:

1. 1 -
   It's functional, so test code coverage actually means something
2. 2 -
   It has typespec support enabling static analysis
3. 3 -
   It's fast as all heck for web workflows
4. 4 -
   It has a low memory footprint and is cheap to deploy
5. 5 -
   It's great at realtime and websockets
6. 6 -
   It's got great opinionated defaults.

Really the only downside is that the community is significantly smaller and so "support" libraries - like those for LLMs aren't as great.

After implementing variations of the exact same glue code in a half dozen projects, I decided to extract those details into a simple, provider agnostic LLM library that could be used for anything from generate requests all the way through Agentic loops with user interaction.

The result is the ALLM library (Agent LLM). The library tries to be as neutral as possible on decisions it doesn't have to make - it doesn't provide things like memory, datastores or the like - but it provides an easy way to implement those things on top of the library. It can work in single-key or SaaS BYO-Key situations. It uses streaming as the primitive, but then provides simple wrappers around streaming for easy sync use cases. It has a first-class testing story with a built-in Fake Provider.

The library provides clear primitives for everything from generation (structured output supported):

{:ok, %ALLM.Response{output_text: text}} =
  ALLM.generate(engine, ALLM.request([ALLM.user("Name three primes.")]))

Through chat multi turn chat with tool calling:

weather =
  ALLM.tool(
    name: "get_weather",
    description: "Return the current weather for a city.",
    schema: %{
      "type" => "object",
      "properties" => %{"city" => %{"type" => "string"}},
      "required" => ["city"]
    },
    handler: fn %{"city" => city} ->
      {:ok, %{forecast: "sunny", city: city}}
    end
  )

engine = ALLM.Engine.put_tools(engine, [weather])

{:ok, result} =
  ALLM.chat(engine, [ALLM.user("What's the weather in Boston?")])

result.final_response.output_text
# => "It's sunny in Boston."

To serializable sessions with state:

session = :erlang.binary_to_term(blob_from_db)

{:ok, session, result} =
  ALLM.Session.reply(engine, session, "What did I just ask?")

session.status
# => :completed
result.final_response.output_text
# => "You asked about the weather in Boston."

The ideas were extracted from 4 different projects that all needed a slightly different take on a LLM library and replaced a few hundred lines of one-off implementations, less robust. Check out the package and the Hexdocs for more info.

ALLM is under heavy development as more and more use cases expand its feature set (a project I was working on just needed an Embedding API) but is now in production and drives, among other things, the data pipelines feeding Unofficial Amesbury