running marimo on modal

I actually posted this to github discussions, so apologies for double-positng. I think this is probably a better forum for this question... (Happy to delete the github one if that's helpful)

I am trying to get marimo running on modal, following https://docs.marimo.io/guides/deploying/programmatically.html and https://docs.marimo.io/guides/deploying/authentication.html

The following code works, up to the point of giving a 401 for /api/kernel/instantiate

Plain Text

import modal
from modal import asgi_app
import marimo

server = marimo.create_asgi_app().with_app(path="", root="/marimo/test.py")

app = modal.App()

@app.function(
    image=modal.Image.debian_slim().pip_install("marimo==0.8.3", "fastapi"),
    gpu=False,
    mounts=[modal.Mount.from_local_dir("./marimo", remote_path="/marimo")],
)
@asgi_app()
def marimo_asgi():
    return server.build()


if __name__ == "__main__":
    modal.serve(app)

I have tried adding middleware and I can see the requests come through.

I can reject requests (return 401) but I can't figure out how to prevent a 401 for /api/kernel/instantiate

Plain Text

    GET / -> 200 OK  (duration: 117.8 ms, execution: 51.9 ms)
    ....
    GET /android-chrome-192x192.png -> 200 OK  (duration: 1.27 s, execution: 0.0 ms)
   POST /api/kernel/instantiate -> 401 Unauthorized  (duration: 1.17 s, execution: 13.1 ms)
CONNECT /ws -> 101 Switching Protocols  (duration: 1.52 s, execution: 0.0 ms)

I think I am missing something fundamental about what's going on here, since I don't see where a token could be validated, or where the 401 is actually happening. I do see marimo-server-token in the header for /api/kernel/instantiate but no authorization or similar.

If anyone has any ideas, I would appreciate it! Running marimo on modal would be fantastic.

26 comments

Here is some example middleware code where I am trying to allow everything (note this is one of several attempts. There is no authorization in the header, but there is marimo-server-token . I can try turning on and off the auth_header check, edit the headers, etc. and no matter how permissive I try to make it, I get the same 401):

Plain Text

def auth_middleware(app):
    async def middleware(scope, receive, send):
        if scope['type'] == 'http':
            headers = dict(scope['headers'])
            auth_header = headers.get(b'authorization', b'').decode()
            expected_auth = f"Bearer {TOKEN}"

            if auth_header != expected_auth:
                print(f"Authentication failed. Expected: {expected_auth}, Got: {auth_header}")
                await send({
                    'type': 'http.response.start',
                    'status': 401,
                    'headers': [(b'content-type', b'text/plain')]
                })
                await send({
                    'type': 'http.response.body',
                    'body': b'Unauthorized\n'
                })
                return
        await app(scope, receive, send)

    return middleware

Thanks for posting here, we monitor Discord more than GitHub discussions. We haven't tried running marimo on modal yet, but it would be very cool ...! Off the top of my head I don't know what the issue is but we can help debug

Thanks! I would be ashamed to admit how much time i've spent trying to get an AI to help figure out authentication here.....

hah, no shame, we've all been there!

also you may be interested that the modal guys are "considering using marimo internally"

https://modal.com/docs/guide/sandbox

Oh, cool! @Eric Zhang at modal was telling us a bit about features that they’re developing that could better support notebooks. Cool to hear that they’re also considering using it for their own work

EEric Zhang

Yes I shared it with some folks in february, it probably won't happen until pyodide gets some kind of grpc support though

EEric Zhang

We're most excited about developing an integrated backend with y'all so you can open Marimo notebooks locally or on the web, but plug into GPUs or other hardware resources in the cloud as desired

EEric Zhang

@btnaughton - are these requests all servered from the same modal container? marimo is stateless, so if /instantiate hits a new container, it will return a 401

i just added concurrency_limit=1, - this fixes the issue you have above. but there seems to still be an issue with marimo running on modal. none of my outputs being streamed back (could be threading or some of the patches we add), we can investigate

maybe its the size of the conatiner - ill try tweaking it

Ah....... thanks! I never would have thought of this.

I am seeing the same thing, some kind of blockage.

I do not see the POST /api/kernel/instantiate request any more for some reason so maybe it's getting stuck here.

Thanks for unsticking me, I'll poke at it more now.

yea, it seems that the /api/kernel/instantiate hangs. i tried to look in the modal logs, but didnt see anything. if you are more familiar with modal, you might be able to find why its hanging. we will also do our own debugging

allow_concurrent_inputs=2,
seems to fix it??

ok as far as i can tell it's working
🎈
thanks!!

Ah sweet! Probs because the ws is always connected

spoke a bit too soon, it dies after a few minutes, with a 401, so i assume it's changing containers or just dropping

i tihnk that should be fixable with some kind of timeout or keep_warm or similar

at some point modal has to disconnect or it's just a server...

We don’t have a concept of resuming a session from a shutdown container. So you need to keep it alive for now.

also i don't see how to edit the code, but i think that's just basic misunderstanding of marimo!

You cannot in run mode - which the asgi api for marimo is.

You can while in edit - but that is through the cli

Editing on a remote server is done via ssh tunneling: https://docs.marimo.io/faq.html#how-do-i-use-marimo-on-a-remote-server

got it -- good to know the asgi app is run only

(I'll update the docs to clarify)

thanks again